{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# 第9章 现代循环神经网络"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 9.1 门控循环单元（GRU） "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.1.1\n",
    "\n",
    "假设我们只想使用时间步 $t'$ 的输入来预测时间步 $t > t'$ 的输出。对于每个时间步，重置门和更新门的最佳值是什么？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;如果只想采用时间步 $t'$ 的输出来预测时间步 $t$ 的输出，则意味着模型需要减少之前时间步在计算中做出的贡献，根据候选隐状态计算公式 9.1.2：  \n",
    "$$\\tilde{H}_t = \\tanh(X_tW_{xh}+(R_t \\odot H_{t-1})W_{hh}+b_r)$$\n",
    "因此需要令**重置门的值趋向于0**，减少过去状态的影响，令模型只考虑当前时间步 $t'$ 所做出的贡献。将重置门设置为零意味着不考虑过去的信息，更专注于当前输入。因此，$R_t \\approx 0$。\n",
    " \n",
    "&emsp;&emsp;在确定候选隐状态后，需要结合更新门进一步确定新的隐状态 $H_t$ 受过去隐状态 $H_{t-1}$ 和候选隐状态 $\\tilde{H}_t$ 的影响程度，而在只考虑时间步 $t'$ 的输入的情况下，需要抛弃过去隐状态 $H_{t-1}$ ，来跳过过去时间步。因此根据最终更新公式 9.1.3：  \n",
    "$$H_t = Z_t \\odot H_{t-1} + (1-Z_t) \\odot \\tilde{H}_t$$\n",
    "需要**更新门的值也趋向于0**，使得新的隐状态更接近于候选隐状态。因此，$Z_t \\approx 0$。\n",
    "\n",
    "&emsp;&emsp;因此，只想使用时间步 $t'$ 的输入来预测时间步 $t$ 的输出时，对于每个时间步，重置门和更新门的最佳值都应为0。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.1.2\n",
    "\n",
    "调整和分析超参数对运行时间、困惑度和输出顺序的影响。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;修改模型超参数，代码及对应结果如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from d2l import torch as d2l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# 定义模型超参数\n",
    "# 超参数内容：[迭代周期，隐藏单元数，批量大小，小批量数据时间步数，学习率]\n",
    "hyper_0 = [500, 512, 35, 32, 1]\n",
    "hyper_1 = [250, 512, 35, 32, 1]\n",
    "hyper_2 = [500, 256, 35, 32, 1]\n",
    "hyper_3 = [500, 512, 10, 32, 1]\n",
    "hyper_4 = [500, 512, 35, 10, 1]\n",
    "hyper_5 = [500, 512, 35, 32, 0.01]\n",
    "hyper_params = [hyper_0, hyper_1, hyper_2, hyper_3, hyper_4, hyper_5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_ch8(net, train_iter, vocab, lr, num_epochs, device,\n",
    "              use_random_iter=False):\n",
    "    \"\"\"\n",
    "    训练模型（定义见第8章）\n",
    "    \"\"\"\n",
    "    # 交叉熵损失函数\n",
    "    loss = nn.CrossEntropyLoss()\n",
    "\n",
    "    # 初始化优化器\n",
    "    if isinstance(net, nn.Module):\n",
    "        updater = torch.optim.SGD(net.parameters(), lr)\n",
    "    else:\n",
    "        updater = lambda batch_size: d2l.sgd(net.params, lr, batch_size)\n",
    "\n",
    "    # 预测函数\n",
    "    predict = lambda prefix: d2l.predict_ch8(prefix, 50, net, vocab, device)\n",
    "\n",
    "    # 训练和预测\n",
    "    for epoch in range(num_epochs):\n",
    "        # 调用train_epoch_ch8函数进行一轮训练\n",
    "        ppl, speed = d2l.train_epoch_ch8(net, train_iter, loss, updater, device, use_random_iter)\n",
    "        \n",
    "        # 每10轮打印一次困惑度\n",
    "        # if (epoch + 1) % 10 == 0:\n",
    "        #     print(f'Epoch {epoch + 1}, Perplexity: {ppl:.1f}')\n",
    "\n",
    "    # 打印一个例子的预测结果\n",
    "    print(predict('time traveller'))\n",
    "\n",
    "    # 返回最终的困惑度值\n",
    "    return ppl\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading ..\\data\\timemachine.txt from http://d2l-data.s3-accelerate.amazonaws.com/timemachine.txt...\n",
      "time traveller came back andfilby s anecdote collapsedthe thing \n",
      "hyper[0]: Timer:17.02, perplexity:1.0\n",
      "time traveller some theprour a moment weich in i arait yous inta\n",
      "hyper[1]: Timer:9.00, perplexity:2.3\n",
      "time traveller proceeded anyreal body must have extension in fou\n",
      "hyper[2]: Timer:18.10, perplexity:1.0\n",
      "time traveller for so it will be convenient to speak of himwas e\n",
      "hyper[3]: Timer:55.82, perplexity:1.0\n",
      "time traveller came back andfilby s anecdote collapsedthe thing \n",
      "hyper[4]: Timer:41.70, perplexity:1.0\n",
      "time traveller   t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t \n",
      "hyper[5]: Timer:18.78, perplexity:16.7\n"
     ]
    }
   ],
   "source": [
    "perplexity = []  # 存储每组超参数训练后的困惑度\n",
    "times = []  # 存储每组超参数训练所花费的时间\n",
    "\n",
    "# 遍历超参数列表\n",
    "for i in range(len(hyper_params)):\n",
    "    num_epochs, num_hiddens, batch_size, num_steps, lr = hyper_params[i]\n",
    "    \n",
    "    # 加载数据和词汇表\n",
    "    train_iter, vocab = d2l.load_data_time_machine(batch_size, num_steps)\n",
    "    \n",
    "    # 创建GRU层和模型\n",
    "    gru_layer = nn.GRU(len(vocab), num_hiddens)\n",
    "    model = d2l.RNNModel(gru_layer, len(vocab))\n",
    "    \n",
    "    # 将模型移动到GPU上（如果可用）\n",
    "    model = model.to(d2l.try_gpu())\n",
    "    \n",
    "    # 创建计时器\n",
    "    timer = d2l.Timer()\n",
    "    timer.start()\n",
    "    \n",
    "    # 训练模型，并获取困惑度值\n",
    "    value = round(train_ch8(model, train_iter, vocab, lr, num_epochs, d2l.try_gpu()), 1)\n",
    "    \n",
    "    # 停止计时并记录时间\n",
    "    time = timer.stop()\n",
    "    \n",
    "    # 将时间和困惑度添加到对应的列表中\n",
    "    times.append(time)\n",
    "    perplexity.append(value)\n",
    "    \n",
    "    # 打印训练结果信息\n",
    "    print('hyper[%d]: Timer:%.2f, perplexity:%.1f' % (i, times[i], perplexity[i]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|迭代周期|隐藏单元数|批量大小|小批量数据时间步数|学习率|困惑度|运行时间|\n",
    "| ----- | ------- | ----- | --------------- | --- | ---- | ----- |\n",
    "|500|512|35|32|1|1.0|17.02|\n",
    "|250|512|35|32|1|2.4|9.00|\n",
    "|500|256|35|32|1|1.0|18.10|\n",
    "|500|512|10|32|1|1.0|55.82|\n",
    "|500|512|35|10|1|1.0|41.7|\n",
    "|500|512|35|32|0.01|16.7|18.78|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;对比hyper[0]中的默认超参数配置，我们可以发现\n",
    "- hyper[1]中缩小迭代周期，导致模型无法完全收敛，导致困惑度增大；\n",
    "- hyper[2]中减少了隐藏层单元数，使得模型在很短时间内完成收敛，并且因为数据集较为简单的原因，困惑度也降低为1；\n",
    "- hyper[3]中减小了批量大小，这使得模型的训练时间开销显著增加；\n",
    "- hyper[4]中将小批量时间步的跨度缩小，虽然也造成了时间开销的增加，但少于批量减少带来的时间增长；\n",
    "- hyper[5]中则对学习率进行了修改，可以发现过低的学习率会使得模型收敛的非常慢，造成困惑度暴涨。\n",
    "\n",
    "&emsp;&emsp;综上可以发现，迭代周期和学习率的缩小会导致模型无法在对应周期内完成收敛，导致模型训练不完全，困惑度显著增加。批量大小即时间步跨度的降低则会导致模型每次加载少量数据，变相造成时间开销的增大；而对于简单的数据，采用一个简单的模型既可以降低时间开销，也可以使得模型很好收敛，确保性能。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.1.3\n",
    "\n",
    "比较`rnn.RNN`和`rnn.GRU`的不同实现对运行时间、困惑度和输出字符串的影响。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from d2l import torch as d2l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_ch8(net, train_iter, vocab, lr, num_epochs, device,\n",
    "              use_random_iter=False):\n",
    "    \"\"\"\n",
    "    训练模型（定义见第8章）\n",
    "    \"\"\"\n",
    "    # 交叉熵损失函数\n",
    "    loss = nn.CrossEntropyLoss()\n",
    "\n",
    "    # 初始化优化器\n",
    "    if isinstance(net, nn.Module):\n",
    "        updater = torch.optim.SGD(net.parameters(), lr)\n",
    "    else:\n",
    "        updater = lambda batch_size: d2l.sgd(net.params, lr, batch_size)\n",
    "\n",
    "    # 预测函数\n",
    "    predict = lambda prefix: d2l.predict_ch8(prefix, 50, net, vocab, device)\n",
    "\n",
    "    # 训练和预测\n",
    "    for epoch in range(num_epochs):\n",
    "        # 调用train_epoch_ch8函数进行一轮训练\n",
    "        ppl, speed = d2l.train_epoch_ch8(net, train_iter, loss, updater, device, use_random_iter)\n",
    "        \n",
    "        # 每10轮打印一次困惑度\n",
    "        if (epoch + 1) % 10 == 0:\n",
    "            print(f'Epoch {epoch + 1}, Perplexity: {ppl:.1f}')\n",
    "\n",
    "    # 打印一个例子的预测结果\n",
    "    print(predict('time traveller'))\n",
    "\n",
    "    # 返回最终的困惑度值\n",
    "    return ppl\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# 设置每个小批次的大小和时间步数\n",
    "batch_size, num_steps = 32, 35\n",
    "\n",
    "# 加载时间机器数据并创建数据迭代器和词汇表\n",
    "train_iter, vocab = d2l.load_data_time_machine(batch_size, num_steps)\n",
    "\n",
    "# 创建计时器对象\n",
    "timer = d2l.Timer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "perplexity 1.3, 341642.0 tokens/sec on cuda:0\n",
      "time traveller but now you begin to seethe object of my investig\n",
      "traveller anthed me time sofilupeomet ees fouprewa d an th \n",
      "time_rnn:19.71\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"252.646875pt\" height=\"183.35625pt\" viewBox=\"0 0 252.646875 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2024-02-14T19:26:52.503440</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 183.35625 \n",
       "L 252.646875 183.35625 \n",
       "L 252.646875 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "L 40.603125 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 76.474554 145.8 \n",
       "L 76.474554 7.2 \n",
       "\" clip-path=\"url(#p539604ecca)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"mfd798a4feb\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mfd798a4feb\" x=\"76.474554\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(66.930804 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 116.331696 145.8 \n",
       "L 116.331696 7.2 \n",
       "\" clip-path=\"url(#p539604ecca)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mfd798a4feb\" x=\"116.331696\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 200 -->\n",
       "      <g transform=\"translate(106.787946 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 156.188839 145.8 \n",
       "L 156.188839 7.2 \n",
       "\" clip-path=\"url(#p539604ecca)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mfd798a4feb\" x=\"156.188839\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 300 -->\n",
       "      <g transform=\"translate(146.645089 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 196.045982 145.8 \n",
       "L 196.045982 7.2 \n",
       "\" clip-path=\"url(#p539604ecca)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mfd798a4feb\" x=\"196.045982\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 400 -->\n",
       "      <g transform=\"translate(186.502232 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "\" clip-path=\"url(#p539604ecca)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mfd798a4feb\" x=\"235.903125\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 500 -->\n",
       "      <g transform=\"translate(226.359375 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(123.025 174.076563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 40.603125 130.329247 \n",
       "L 235.903125 130.329247 \n",
       "\" clip-path=\"url(#p539604ecca)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"md11519be79\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#md11519be79\" x=\"40.603125\" y=\"130.329247\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(27.240625 134.128466)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 40.603125 105.340436 \n",
       "L 235.903125 105.340436 \n",
       "\" clip-path=\"url(#p539604ecca)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#md11519be79\" x=\"40.603125\" y=\"105.340436\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 4 -->\n",
       "      <g transform=\"translate(27.240625 109.139655)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 40.603125 80.351626 \n",
       "L 235.903125 80.351626 \n",
       "\" clip-path=\"url(#p539604ecca)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#md11519be79\" x=\"40.603125\" y=\"80.351626\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 6 -->\n",
       "      <g transform=\"translate(27.240625 84.150845)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n",
       "Q 1688 2584 1439 2293 \n",
       "Q 1191 2003 1191 1497 \n",
       "Q 1191 994 1439 701 \n",
       "Q 1688 409 2113 409 \n",
       "Q 2538 409 2786 701 \n",
       "Q 3034 994 3034 1497 \n",
       "Q 3034 2003 2786 2293 \n",
       "Q 2538 2584 2113 2584 \n",
       "z\n",
       "M 3366 4563 \n",
       "L 3366 3988 \n",
       "Q 3128 4100 2886 4159 \n",
       "Q 2644 4219 2406 4219 \n",
       "Q 1781 4219 1451 3797 \n",
       "Q 1122 3375 1075 2522 \n",
       "Q 1259 2794 1537 2939 \n",
       "Q 1816 3084 2150 3084 \n",
       "Q 2853 3084 3261 2657 \n",
       "Q 3669 2231 3669 1497 \n",
       "Q 3669 778 3244 343 \n",
       "Q 2819 -91 2113 -91 \n",
       "Q 1303 -91 875 529 \n",
       "Q 447 1150 447 2328 \n",
       "Q 447 3434 972 4092 \n",
       "Q 1497 4750 2381 4750 \n",
       "Q 2619 4750 2861 4703 \n",
       "Q 3103 4656 3366 4563 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <path d=\"M 40.603125 55.362816 \n",
       "L 235.903125 55.362816 \n",
       "\" clip-path=\"url(#p539604ecca)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#md11519be79\" x=\"40.603125\" y=\"55.362816\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 8 -->\n",
       "      <g transform=\"translate(27.240625 59.162034)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n",
       "Q 1584 2216 1326 1975 \n",
       "Q 1069 1734 1069 1313 \n",
       "Q 1069 891 1326 650 \n",
       "Q 1584 409 2034 409 \n",
       "Q 2484 409 2743 651 \n",
       "Q 3003 894 3003 1313 \n",
       "Q 3003 1734 2745 1975 \n",
       "Q 2488 2216 2034 2216 \n",
       "z\n",
       "M 1403 2484 \n",
       "Q 997 2584 770 2862 \n",
       "Q 544 3141 544 3541 \n",
       "Q 544 4100 942 4425 \n",
       "Q 1341 4750 2034 4750 \n",
       "Q 2731 4750 3128 4425 \n",
       "Q 3525 4100 3525 3541 \n",
       "Q 3525 3141 3298 2862 \n",
       "Q 3072 2584 2669 2484 \n",
       "Q 3125 2378 3379 2068 \n",
       "Q 3634 1759 3634 1313 \n",
       "Q 3634 634 3220 271 \n",
       "Q 2806 -91 2034 -91 \n",
       "Q 1263 -91 848 271 \n",
       "Q 434 634 434 1313 \n",
       "Q 434 1759 690 2068 \n",
       "Q 947 2378 1403 2484 \n",
       "z\n",
       "M 1172 3481 \n",
       "Q 1172 3119 1398 2916 \n",
       "Q 1625 2713 2034 2713 \n",
       "Q 2441 2713 2670 2916 \n",
       "Q 2900 3119 2900 3481 \n",
       "Q 2900 3844 2670 4047 \n",
       "Q 2441 4250 2034 4250 \n",
       "Q 1625 4250 1398 4047 \n",
       "Q 1172 3844 1172 3481 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-38\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <path d=\"M 40.603125 30.374005 \n",
       "L 235.903125 30.374005 \n",
       "\" clip-path=\"url(#p539604ecca)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#md11519be79\" x=\"40.603125\" y=\"30.374005\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(20.878125 34.173224)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_12\">\n",
       "     <!-- perplexity -->\n",
       "     <g transform=\"translate(14.798437 101.626563)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"229.589844\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"257.373047\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"317.146484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"376.326172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"404.109375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"443.318359\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_21\">\n",
       "    <path d=\"M 40.603125 13.5 \n",
       "L 44.588839 40.75874 \n",
       "L 48.574554 49.877368 \n",
       "L 52.560268 58.10768 \n",
       "L 56.545982 62.733938 \n",
       "L 60.531696 68.344606 \n",
       "L 64.517411 78.527081 \n",
       "L 68.503125 86.142062 \n",
       "L 72.488839 98.853621 \n",
       "L 76.474554 108.01121 \n",
       "L 80.460268 114.354187 \n",
       "L 84.445982 120.169012 \n",
       "L 88.431696 122.850311 \n",
       "L 92.417411 127.638137 \n",
       "L 96.403125 129.585314 \n",
       "L 100.388839 130.915325 \n",
       "L 104.374554 132.66893 \n",
       "L 108.360268 133.701055 \n",
       "L 112.345982 134.178436 \n",
       "L 116.331696 135.278727 \n",
       "L 120.317411 135.01906 \n",
       "L 124.303125 136.114248 \n",
       "L 128.288839 136.580343 \n",
       "L 132.274554 136.805757 \n",
       "L 136.260268 136.64814 \n",
       "L 140.245982 137.831434 \n",
       "L 144.231696 137.730328 \n",
       "L 148.217411 137.568719 \n",
       "L 152.203125 137.83878 \n",
       "L 156.188839 138.267312 \n",
       "L 160.174554 137.789641 \n",
       "L 164.160268 138.292143 \n",
       "L 168.145982 137.119465 \n",
       "L 172.131696 137.613826 \n",
       "L 176.117411 138.758465 \n",
       "L 180.103125 138.146788 \n",
       "L 184.088839 138.60358 \n",
       "L 188.074554 138.914084 \n",
       "L 192.060268 138.693757 \n",
       "L 196.045982 138.514324 \n",
       "L 200.031696 138.26124 \n",
       "L 204.017411 138.752059 \n",
       "L 208.003125 138.635992 \n",
       "L 211.988839 138.165013 \n",
       "L 215.974554 139.155762 \n",
       "L 219.960268 138.666193 \n",
       "L 223.945982 139.5 \n",
       "L 227.931696 138.315729 \n",
       "L 231.917411 139.248469 \n",
       "L 235.903125 139.101214 \n",
       "\" clip-path=\"url(#p539604ecca)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 40.603125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 235.903125 145.8 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 40.603125 7.2 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 173.628125 29.878125 \n",
       "L 228.903125 29.878125 \n",
       "Q 230.903125 29.878125 230.903125 27.878125 \n",
       "L 230.903125 14.2 \n",
       "Q 230.903125 12.2 228.903125 12.2 \n",
       "L 173.628125 12.2 \n",
       "Q 171.628125 12.2 171.628125 14.2 \n",
       "L 171.628125 27.878125 \n",
       "Q 171.628125 29.878125 173.628125 29.878125 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_22\">\n",
       "     <path d=\"M 175.628125 20.298438 \n",
       "L 185.628125 20.298438 \n",
       "L 195.628125 20.298438 \n",
       "\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- train -->\n",
       "     <g transform=\"translate(203.628125 23.798438)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p539604ecca\">\n",
       "   <rect x=\"40.603125\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 350x250 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# RNN\n",
    "# 设置RNN的隐藏单元数\n",
    "num_hiddens = 256\n",
    "\n",
    "# 创建RNN层，指定输入大小为词汇表大小，隐藏单元数为num_hiddens\n",
    "rnn_layer = nn.RNN(len(vocab), num_hiddens)\n",
    "\n",
    "# 尝试使用GPU，如果可用的话\n",
    "device = d2l.try_gpu()\n",
    "\n",
    "# 创建RNN模型，包含上面创建的RNN层，指定词汇表大小\n",
    "net = d2l.RNNModel(rnn_layer, vocab_size=len(vocab))\n",
    "\n",
    "# 将模型移动到指定设备（GPU或CPU）\n",
    "net = net.to(device)\n",
    "\n",
    "# 设置训练的超参数\n",
    "num_epochs, lr = 500, 1\n",
    "\n",
    "# 创建计时器对象并开始计时\n",
    "timer.start()\n",
    "\n",
    "# 训练RNN模型，使用指定的数据迭代器、词汇表、学习率等\n",
    "d2l.train_ch8(net, train_iter, vocab, lr, num_epochs, device)\n",
    "\n",
    "# 停止计时并记录时间\n",
    "time_rnn = timer.stop()\n",
    "\n",
    "# 打印训练时间\n",
    "print('time_rnn:%.2f' % time_rnn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "perplexity 1.1, 224634.9 tokens/sec on cuda:0\n",
      "time traveller for so it will mern acd merearen do not the sitio\n",
      "travelleryou can show black is white by argument said filby\n",
      "time_gru:25.25\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"252.646875pt\" height=\"183.35625pt\" viewBox=\"0 0 252.646875 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2024-02-14T19:27:17.822144</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 183.35625 \n",
       "L 252.646875 183.35625 \n",
       "L 252.646875 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "L 40.603125 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 76.474554 145.8 \n",
       "L 76.474554 7.2 \n",
       "\" clip-path=\"url(#pacd91079e5)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"m84695ff1a6\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m84695ff1a6\" x=\"76.474554\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(66.930804 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 116.331696 145.8 \n",
       "L 116.331696 7.2 \n",
       "\" clip-path=\"url(#pacd91079e5)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m84695ff1a6\" x=\"116.331696\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 200 -->\n",
       "      <g transform=\"translate(106.787946 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 156.188839 145.8 \n",
       "L 156.188839 7.2 \n",
       "\" clip-path=\"url(#pacd91079e5)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m84695ff1a6\" x=\"156.188839\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 300 -->\n",
       "      <g transform=\"translate(146.645089 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 196.045982 145.8 \n",
       "L 196.045982 7.2 \n",
       "\" clip-path=\"url(#pacd91079e5)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m84695ff1a6\" x=\"196.045982\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 400 -->\n",
       "      <g transform=\"translate(186.502232 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "\" clip-path=\"url(#pacd91079e5)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m84695ff1a6\" x=\"235.903125\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 500 -->\n",
       "      <g transform=\"translate(226.359375 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(123.025 174.076563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 40.603125 105.225814 \n",
       "L 235.903125 105.225814 \n",
       "\" clip-path=\"url(#pacd91079e5)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"m8a72360fa7\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8a72360fa7\" x=\"40.603125\" y=\"105.225814\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(27.240625 109.025032)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 40.603125 62.022458 \n",
       "L 235.903125 62.022458 \n",
       "\" clip-path=\"url(#pacd91079e5)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8a72360fa7\" x=\"40.603125\" y=\"62.022458\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(20.878125 65.821676)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 40.603125 18.819102 \n",
       "L 235.903125 18.819102 \n",
       "\" clip-path=\"url(#pacd91079e5)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8a72360fa7\" x=\"40.603125\" y=\"18.819102\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 15 -->\n",
       "      <g transform=\"translate(20.878125 22.618321)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_10\">\n",
       "     <!-- perplexity -->\n",
       "     <g transform=\"translate(14.798437 101.626563)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"229.589844\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"257.373047\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"317.146484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"376.326172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"404.109375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"443.318359\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_17\">\n",
       "    <path d=\"M 40.603125 13.5 \n",
       "L 44.588839 41.805347 \n",
       "L 48.574554 55.444669 \n",
       "L 52.560268 61.705379 \n",
       "L 56.545982 64.811096 \n",
       "L 60.531696 69.20687 \n",
       "L 64.517411 73.038244 \n",
       "L 68.503125 75.692303 \n",
       "L 72.488839 78.941104 \n",
       "L 76.474554 81.654284 \n",
       "L 80.460268 84.589064 \n",
       "L 84.445982 88.093341 \n",
       "L 88.431696 90.382069 \n",
       "L 92.417411 93.360151 \n",
       "L 96.403125 96.377476 \n",
       "L 100.388839 100.506958 \n",
       "L 104.374554 103.189844 \n",
       "L 108.360268 107.661352 \n",
       "L 112.345982 112.528258 \n",
       "L 116.331696 115.444153 \n",
       "L 120.317411 120.27552 \n",
       "L 124.303125 124.903476 \n",
       "L 128.288839 128.981298 \n",
       "L 132.274554 132.360254 \n",
       "L 136.260268 134.638827 \n",
       "L 140.245982 135.956496 \n",
       "L 144.231696 137.656023 \n",
       "L 148.217411 138.061062 \n",
       "L 152.203125 138.573754 \n",
       "L 156.188839 138.777443 \n",
       "L 160.174554 138.915784 \n",
       "L 164.160268 138.933128 \n",
       "L 168.145982 138.208881 \n",
       "L 172.131696 139.085146 \n",
       "L 176.117411 138.716568 \n",
       "L 180.103125 139.29918 \n",
       "L 184.088839 139.175502 \n",
       "L 188.074554 139.239324 \n",
       "L 192.060268 139.124332 \n",
       "L 196.045982 139.214551 \n",
       "L 200.031696 139.304989 \n",
       "L 204.017411 139.346396 \n",
       "L 208.003125 139.39294 \n",
       "L 211.988839 139.316512 \n",
       "L 215.974554 139.410683 \n",
       "L 219.960268 139.434425 \n",
       "L 223.945982 139.461111 \n",
       "L 227.931696 139.463054 \n",
       "L 231.917411 139.5 \n",
       "L 235.903125 139.304421 \n",
       "\" clip-path=\"url(#pacd91079e5)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 40.603125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 235.903125 145.8 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 40.603125 7.2 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 173.628125 29.878125 \n",
       "L 228.903125 29.878125 \n",
       "Q 230.903125 29.878125 230.903125 27.878125 \n",
       "L 230.903125 14.2 \n",
       "Q 230.903125 12.2 228.903125 12.2 \n",
       "L 173.628125 12.2 \n",
       "Q 171.628125 12.2 171.628125 14.2 \n",
       "L 171.628125 27.878125 \n",
       "Q 171.628125 29.878125 173.628125 29.878125 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_18\">\n",
       "     <path d=\"M 175.628125 20.298438 \n",
       "L 185.628125 20.298438 \n",
       "L 195.628125 20.298438 \n",
       "\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    </g>\n",
       "    <g id=\"text_11\">\n",
       "     <!-- train -->\n",
       "     <g transform=\"translate(203.628125 23.798438)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pacd91079e5\">\n",
       "   <rect x=\"40.603125\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 350x250 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 获取词汇表的大小\n",
    "num_inputs = len(vocab)\n",
    "\n",
    "# 创建GRU层，指定输入大小为词汇表大小，隐藏单元数为num_hiddens\n",
    "gru_layer = nn.GRU(num_inputs, num_hiddens)\n",
    "\n",
    "# 创建GRU模型，包含上面创建的GRU层，指定词汇表大小\n",
    "model = d2l.RNNModel(gru_layer, len(vocab))\n",
    "\n",
    "# 将模型移动到指定设备（GPU或CPU）\n",
    "model = model.to(device)\n",
    "\n",
    "# 创建计时器对象并开始计时\n",
    "timer.start()\n",
    "\n",
    "# 训练GRU模型，使用指定的数据迭代器、词汇表、学习率等\n",
    "d2l.train_ch8(model, train_iter, vocab, lr, num_epochs, device)\n",
    "\n",
    "# 停止计时并记录时间\n",
    "time_gru = timer.stop()\n",
    "\n",
    "# 打印训练时间\n",
    "print('time_gru:%.2f' % time_gru)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|模型|困惑度|运行时间|\n",
    "| -- | --- | ----- |\n",
    "|RNN|1.3|19.71|\n",
    "|GRU|1.1|25.25|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;相较于RNN，GRU在重置门和更新门的计算，以及计算候选隐状态和确定最终隐状态的过程中增加了大量矩阵运算，这些新增的矩阵运算会使得模型的训练时间出现增加。但是得益于GRU中重置门与更新门的结构优势，GRU生成结果的困惑度仅为1.0，低于RNN中的1.3困惑度，输出字符串也更好。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.1.4\n",
    "\n",
    "如果仅仅实现门控循环单元的一部分，例如，只有一个重置门或一个更新门会怎样？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from d2l import torch as d2l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_ch8(net, train_iter, vocab, lr, num_epochs, device,\n",
    "              use_random_iter=False):\n",
    "    \"\"\"\n",
    "    训练模型（定义见第8章）\n",
    "    \"\"\"\n",
    "    # 交叉熵损失函数\n",
    "    loss = nn.CrossEntropyLoss()\n",
    "\n",
    "    # 初始化优化器\n",
    "    if isinstance(net, nn.Module):\n",
    "        updater = torch.optim.SGD(net.parameters(), lr)\n",
    "    else:\n",
    "        updater = lambda batch_size: d2l.sgd(net.params, lr, batch_size)\n",
    "\n",
    "    # 预测函数\n",
    "    predict = lambda prefix: d2l.predict_ch8(prefix, 50, net, vocab, device)\n",
    "\n",
    "    # 训练和预测\n",
    "    for epoch in range(num_epochs):\n",
    "        # 调用train_epoch_ch8函数进行一轮训练\n",
    "        ppl, speed = d2l.train_epoch_ch8(net, train_iter, loss, updater, device, use_random_iter)\n",
    "        \n",
    "        # 每10轮打印一次困惑度\n",
    "        if (epoch + 1) % 10 == 0:\n",
    "            print(f'Epoch {epoch + 1}, Perplexity: {ppl:.1f}')\n",
    "\n",
    "    # 打印一个例子的预测结果\n",
    "    print(predict('time traveller'))\n",
    "\n",
    "    # 返回最终的困惑度值\n",
    "    return ppl\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# 仅重置门\n",
    "def get_params_R(vocab_size, num_hiddens, device):\n",
    "    # 设置输入和输出的数量\n",
    "    num_inputs = num_outputs = vocab_size\n",
    "\n",
    "    # 定义用于初始化参数的函数\n",
    "    def normal(shape):\n",
    "        return torch.randn(size=shape, device=device) * 0.01\n",
    "\n",
    "    # 定义一个辅助函数，用于初始化三个参数的元组\n",
    "    def three():\n",
    "        return (normal((num_inputs, num_hiddens)),\n",
    "                normal((num_hiddens, num_hiddens)),\n",
    "                torch.zeros(num_hiddens, device=device))\n",
    "\n",
    "    # 初始化重置门参数\n",
    "    W_xr, W_hr, b_r = three()  # 重置门参数\n",
    "\n",
    "    # 初始化候选隐状态参数\n",
    "    W_xh, W_hh, b_h = three()  # 候选隐状态参数\n",
    "\n",
    "    # 初始化输出层参数\n",
    "    W_hq = normal((num_hiddens, num_outputs))\n",
    "    b_q = torch.zeros(num_outputs, device=device)\n",
    "\n",
    "    # 将所有参数放在一个列表中\n",
    "    params = [W_xr, W_hr, b_r, W_xh, W_hh, b_h, W_hq, b_q]\n",
    "\n",
    "    # 将梯度信息添加到所有参数中\n",
    "    for param in params:\n",
    "        param.requires_grad_(True)\n",
    "\n",
    "    return params\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def init_gru_state(batch_size, num_hiddens, device):\n",
    "    # 初始化GRU状态，返回一个元组，包含一个形状为(batch_size, num_hiddens)的零张量\n",
    "    return (torch.zeros((batch_size, num_hiddens), device=device), )\n",
    "\n",
    "def gru(inputs, state, params):\n",
    "    # 获取GRU的所有参数\n",
    "    W_xr, W_hr, b_r, W_xh, W_hh, b_h, W_hq, b_q = params\n",
    "    \n",
    "    # 从状态中获取隐层的值\n",
    "    H, = state\n",
    "    \n",
    "    # 存储所有时间步的输出\n",
    "    outputs = []\n",
    "    \n",
    "    # 遍历所有时间步\n",
    "    for X in inputs:\n",
    "        # Z = torch.sigmoid((X @ W_xz) + (H @ W_hz) + b_z)\n",
    "        # 计算重置门\n",
    "        R = torch.sigmoid((X @ W_xr) + (H @ W_hr) + b_r)\n",
    "        \n",
    "        # 计算候选隐状态\n",
    "        H_tilda = torch.tanh((X @ W_xh) + ((R * H) @ W_hh) + b_h)\n",
    "        \n",
    "        # 更新隐状态\n",
    "        H = H_tilda\n",
    "        \n",
    "        # 计算当前时间步的输出\n",
    "        Y = H @ W_hq + b_q\n",
    "        \n",
    "        # 将输出添加到列表中\n",
    "        outputs.append(Y)\n",
    "    \n",
    "    # 沿着时间步连接所有输出，并返回新的状态\n",
    "    return torch.cat(outputs, dim=0), (H,)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "perplexity 1.2, 61112.7 tokens/sec on cuda:0\n",
      "time travelleryou can show black is white by argument said filby\n",
      "travelleryou can show black as alwaylofilby explacating im \n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"252.646875pt\" height=\"183.35625pt\" viewBox=\"0 0 252.646875 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2024-02-14T19:30:08.805557</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 183.35625 \n",
       "L 252.646875 183.35625 \n",
       "L 252.646875 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "L 40.603125 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 76.474554 145.8 \n",
       "L 76.474554 7.2 \n",
       "\" clip-path=\"url(#p0bd8b9005c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"m8b53fba3cf\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8b53fba3cf\" x=\"76.474554\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(66.930804 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 116.331696 145.8 \n",
       "L 116.331696 7.2 \n",
       "\" clip-path=\"url(#p0bd8b9005c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8b53fba3cf\" x=\"116.331696\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 200 -->\n",
       "      <g transform=\"translate(106.787946 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 156.188839 145.8 \n",
       "L 156.188839 7.2 \n",
       "\" clip-path=\"url(#p0bd8b9005c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8b53fba3cf\" x=\"156.188839\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 300 -->\n",
       "      <g transform=\"translate(146.645089 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 196.045982 145.8 \n",
       "L 196.045982 7.2 \n",
       "\" clip-path=\"url(#p0bd8b9005c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8b53fba3cf\" x=\"196.045982\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 400 -->\n",
       "      <g transform=\"translate(186.502232 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "\" clip-path=\"url(#p0bd8b9005c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8b53fba3cf\" x=\"235.903125\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 500 -->\n",
       "      <g transform=\"translate(226.359375 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(123.025 174.076563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 40.603125 103.726912 \n",
       "L 235.903125 103.726912 \n",
       "\" clip-path=\"url(#p0bd8b9005c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"m14c84f1c45\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m14c84f1c45\" x=\"40.603125\" y=\"103.726912\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(27.240625 107.52613)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 40.603125 58.633222 \n",
       "L 235.903125 58.633222 \n",
       "\" clip-path=\"url(#p0bd8b9005c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m14c84f1c45\" x=\"40.603125\" y=\"58.633222\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(20.878125 62.432441)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 40.603125 13.539533 \n",
       "L 235.903125 13.539533 \n",
       "\" clip-path=\"url(#p0bd8b9005c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m14c84f1c45\" x=\"40.603125\" y=\"13.539533\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 15 -->\n",
       "      <g transform=\"translate(20.878125 17.338752)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_10\">\n",
       "     <!-- perplexity -->\n",
       "     <g transform=\"translate(14.798437 101.626563)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"229.589844\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"257.373047\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"317.146484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"376.326172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"404.109375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"443.318359\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_17\">\n",
       "    <path d=\"M 40.603125 13.5 \n",
       "L 44.588839 47.120641 \n",
       "L 48.574554 55.576845 \n",
       "L 52.560268 61.923061 \n",
       "L 56.545982 65.705052 \n",
       "L 60.531696 67.675298 \n",
       "L 64.517411 70.541136 \n",
       "L 68.503125 72.731194 \n",
       "L 72.488839 74.8939 \n",
       "L 76.474554 76.478908 \n",
       "L 80.460268 78.133703 \n",
       "L 84.445982 81.549757 \n",
       "L 88.431696 83.686749 \n",
       "L 92.417411 86.618408 \n",
       "L 96.403125 88.819911 \n",
       "L 100.388839 93.992329 \n",
       "L 104.374554 97.061909 \n",
       "L 108.360268 99.838693 \n",
       "L 112.345982 103.784483 \n",
       "L 116.331696 109.140295 \n",
       "L 120.317411 113.352425 \n",
       "L 124.303125 118.359694 \n",
       "L 128.288839 122.865441 \n",
       "L 132.274554 126.170667 \n",
       "L 136.260268 129.824914 \n",
       "L 140.245982 133.064656 \n",
       "L 144.231696 135.517331 \n",
       "L 148.217411 137.409438 \n",
       "L 152.203125 138.297818 \n",
       "L 156.188839 136.80723 \n",
       "L 160.174554 138.773759 \n",
       "L 164.160268 138.922017 \n",
       "L 168.145982 138.885468 \n",
       "L 172.131696 139.10496 \n",
       "L 176.117411 137.507979 \n",
       "L 180.103125 139.229593 \n",
       "L 184.088839 139.320864 \n",
       "L 188.074554 139.159048 \n",
       "L 192.060268 139.382236 \n",
       "L 196.045982 139.42509 \n",
       "L 200.031696 139.07781 \n",
       "L 204.017411 139.342993 \n",
       "L 208.003125 139.424591 \n",
       "L 211.988839 139.309858 \n",
       "L 215.974554 139.463588 \n",
       "L 219.960268 139.315356 \n",
       "L 223.945982 139.335996 \n",
       "L 227.931696 139.485288 \n",
       "L 231.917411 139.5 \n",
       "L 235.903125 137.828192 \n",
       "\" clip-path=\"url(#p0bd8b9005c)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 40.603125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 235.903125 145.8 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 40.603125 7.2 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 173.628125 29.878125 \n",
       "L 228.903125 29.878125 \n",
       "Q 230.903125 29.878125 230.903125 27.878125 \n",
       "L 230.903125 14.2 \n",
       "Q 230.903125 12.2 228.903125 12.2 \n",
       "L 173.628125 12.2 \n",
       "Q 171.628125 12.2 171.628125 14.2 \n",
       "L 171.628125 27.878125 \n",
       "Q 171.628125 29.878125 173.628125 29.878125 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_18\">\n",
       "     <path d=\"M 175.628125 20.298438 \n",
       "L 185.628125 20.298438 \n",
       "L 195.628125 20.298438 \n",
       "\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    </g>\n",
       "    <g id=\"text_11\">\n",
       "     <!-- train -->\n",
       "     <g transform=\"translate(203.628125 23.798438)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p0bd8b9005c\">\n",
       "   <rect x=\"40.603125\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 350x250 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 设置词汇表大小、隐藏单元数和设备（如果可用则使用GPU）\n",
    "vocab_size, num_hiddens, device = len(vocab), 256, d2l.try_gpu()\n",
    "\n",
    "# 设置训练轮数和学习率\n",
    "num_epochs, lr = 500, 1\n",
    "\n",
    "# 使用GRU从头开始创建一个RNN模型\n",
    "model = d2l.RNNModelScratch(len(vocab), num_hiddens, device, get_params_R,\n",
    "                            init_gru_state, gru)\n",
    "\n",
    "# 使用提供的数据迭代器、词汇表、学习率等训练模型\n",
    "d2l.train_ch8(model, train_iter, vocab, lr, num_epochs, device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# 仅更新门\n",
    "def get_params_Z(vocab_size, num_hiddens, device):\n",
    "    # 设置输入和输出的数量\n",
    "    num_inputs = num_outputs = vocab_size\n",
    "\n",
    "    # 定义用于初始化参数的函数\n",
    "    def normal(shape):\n",
    "        return torch.randn(size=shape, device=device) * 0.01\n",
    "\n",
    "    # 定义一个辅助函数，用于初始化三个参数的元组\n",
    "    def three():\n",
    "        return (normal((num_inputs, num_hiddens)),\n",
    "                normal((num_hiddens, num_hiddens)),\n",
    "                torch.zeros(num_hiddens, device=device))\n",
    "\n",
    "    # 初始化更新门参数\n",
    "    W_xz, W_hz, b_z = three()  # 更新门参数\n",
    "\n",
    "    # 初始化候选隐状态参数\n",
    "    W_xh, W_hh, b_h = three()  # 候选隐状态参数\n",
    "\n",
    "    # 初始化输出层参数\n",
    "    W_hq = normal((num_hiddens, num_outputs))\n",
    "    b_q = torch.zeros(num_outputs, device=device)\n",
    "\n",
    "    # 将所有参数放在一个列表中\n",
    "    params = [W_xz, W_hz, b_z, W_xh, W_hh, b_h, W_hq, b_q]\n",
    "\n",
    "    # 将梯度信息添加到所有参数中\n",
    "    for param in params:\n",
    "        param.requires_grad_(True)\n",
    "\n",
    "    return params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def init_gru_state(batch_size, num_hiddens, device):\n",
    "    # 初始化GRU状态，返回一个元组，包含一个形状为(batch_size, num_hiddens)的零张量\n",
    "    return (torch.zeros((batch_size, num_hiddens), device=device), )\n",
    "\n",
    "def gru(inputs, state, params):\n",
    "    # 获取GRU的所有参数\n",
    "    W_xz, W_hz, b_z, W_xh, W_hh, b_h, W_hq, b_q = params\n",
    "    \n",
    "    # 从状态中获取隐层的值\n",
    "    H, = state\n",
    "    \n",
    "    # 存储所有时间步的输出\n",
    "    outputs = []\n",
    "    \n",
    "    # 遍历所有时间步\n",
    "    for X in inputs:\n",
    "        # 计算更新门\n",
    "        Z = torch.sigmoid((X @ W_xz) + (H @ W_hz) + b_z)\n",
    "        \n",
    "        # 计算候选隐状态\n",
    "        H_tilda = torch.tanh((X @ W_xh) + (H @ W_hh) + b_h)\n",
    "        \n",
    "        # 更新隐状态\n",
    "        H = Z * H + (1 - Z) * H_tilda\n",
    "        \n",
    "        # 计算当前时间步的输出\n",
    "        Y = H @ W_hq + b_q\n",
    "        \n",
    "        # 将输出添加到列表中\n",
    "        outputs.append(Y)\n",
    "    \n",
    "    # 沿着时间步连接所有输出，并返回新的状态\n",
    "    return torch.cat(outputs, dim=0), (H,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "perplexity 1.1, 55200.0 tokens/sec on cuda:0\n",
      "time travelleryou can show black is white by argument said filby\n",
      "travelleryou can show black is white by argument said filby\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"252.646875pt\" height=\"183.35625pt\" viewBox=\"0 0 252.646875 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2024-02-14T19:31:44.028868</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 183.35625 \n",
       "L 252.646875 183.35625 \n",
       "L 252.646875 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "L 40.603125 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 76.474554 145.8 \n",
       "L 76.474554 7.2 \n",
       "\" clip-path=\"url(#p113f4dfbbc)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"m502e730d92\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m502e730d92\" x=\"76.474554\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(66.930804 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 116.331696 145.8 \n",
       "L 116.331696 7.2 \n",
       "\" clip-path=\"url(#p113f4dfbbc)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m502e730d92\" x=\"116.331696\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 200 -->\n",
       "      <g transform=\"translate(106.787946 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 156.188839 145.8 \n",
       "L 156.188839 7.2 \n",
       "\" clip-path=\"url(#p113f4dfbbc)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m502e730d92\" x=\"156.188839\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 300 -->\n",
       "      <g transform=\"translate(146.645089 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 196.045982 145.8 \n",
       "L 196.045982 7.2 \n",
       "\" clip-path=\"url(#p113f4dfbbc)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m502e730d92\" x=\"196.045982\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 400 -->\n",
       "      <g transform=\"translate(186.502232 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "\" clip-path=\"url(#p113f4dfbbc)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m502e730d92\" x=\"235.903125\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 500 -->\n",
       "      <g transform=\"translate(226.359375 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(123.025 174.076563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 40.603125 108.854231 \n",
       "L 235.903125 108.854231 \n",
       "\" clip-path=\"url(#p113f4dfbbc)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"m964901f352\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m964901f352\" x=\"40.603125\" y=\"108.854231\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(27.240625 112.65345)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 40.603125 70.199867 \n",
       "L 235.903125 70.199867 \n",
       "\" clip-path=\"url(#p113f4dfbbc)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m964901f352\" x=\"40.603125\" y=\"70.199867\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(20.878125 73.999086)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 40.603125 31.545503 \n",
       "L 235.903125 31.545503 \n",
       "\" clip-path=\"url(#p113f4dfbbc)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m964901f352\" x=\"40.603125\" y=\"31.545503\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 15 -->\n",
       "      <g transform=\"translate(20.878125 35.344721)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_10\">\n",
       "     <!-- perplexity -->\n",
       "     <g transform=\"translate(14.798437 101.626563)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"229.589844\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"257.373047\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"317.146484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"376.326172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"404.109375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"443.318359\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_17\">\n",
       "    <path d=\"M 40.603125 13.5 \n",
       "L 44.588839 27.00462 \n",
       "L 48.574554 50.144237 \n",
       "L 52.560268 60.72657 \n",
       "L 56.545982 65.56949 \n",
       "L 60.531696 70.429545 \n",
       "L 64.517411 73.385889 \n",
       "L 68.503125 76.422012 \n",
       "L 72.488839 79.620635 \n",
       "L 76.474554 83.248716 \n",
       "L 80.460268 85.940644 \n",
       "L 84.445982 89.373596 \n",
       "L 88.431696 91.70432 \n",
       "L 92.417411 95.222258 \n",
       "L 96.403125 97.598624 \n",
       "L 100.388839 100.469128 \n",
       "L 104.374554 105.178323 \n",
       "L 108.360268 108.778336 \n",
       "L 112.345982 111.540122 \n",
       "L 116.331696 115.032887 \n",
       "L 120.317411 119.896246 \n",
       "L 124.303125 123.942956 \n",
       "L 128.288839 127.399623 \n",
       "L 132.274554 129.721909 \n",
       "L 136.260268 132.963642 \n",
       "L 140.245982 135.344289 \n",
       "L 144.231696 136.57528 \n",
       "L 148.217411 137.192723 \n",
       "L 152.203125 138.375686 \n",
       "L 156.188839 138.547515 \n",
       "L 160.174554 138.854196 \n",
       "L 164.160268 138.973109 \n",
       "L 168.145982 137.963518 \n",
       "L 172.131696 139.226698 \n",
       "L 176.117411 139.057613 \n",
       "L 180.103125 139.076716 \n",
       "L 184.088839 139.116263 \n",
       "L 188.074554 139.328661 \n",
       "L 192.060268 139.421001 \n",
       "L 196.045982 137.717116 \n",
       "L 200.031696 139.230035 \n",
       "L 204.017411 139.394203 \n",
       "L 208.003125 139.409738 \n",
       "L 211.988839 139.434542 \n",
       "L 215.974554 139.37698 \n",
       "L 219.960268 139.303727 \n",
       "L 223.945982 139.5 \n",
       "L 227.931696 139.458543 \n",
       "L 231.917411 139.479328 \n",
       "L 235.903125 139.36665 \n",
       "\" clip-path=\"url(#p113f4dfbbc)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 40.603125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 235.903125 145.8 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 40.603125 7.2 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 173.628125 29.878125 \n",
       "L 228.903125 29.878125 \n",
       "Q 230.903125 29.878125 230.903125 27.878125 \n",
       "L 230.903125 14.2 \n",
       "Q 230.903125 12.2 228.903125 12.2 \n",
       "L 173.628125 12.2 \n",
       "Q 171.628125 12.2 171.628125 14.2 \n",
       "L 171.628125 27.878125 \n",
       "Q 171.628125 29.878125 173.628125 29.878125 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_18\">\n",
       "     <path d=\"M 175.628125 20.298438 \n",
       "L 185.628125 20.298438 \n",
       "L 195.628125 20.298438 \n",
       "\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    </g>\n",
       "    <g id=\"text_11\">\n",
       "     <!-- train -->\n",
       "     <g transform=\"translate(203.628125 23.798438)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p113f4dfbbc\">\n",
       "   <rect x=\"40.603125\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 350x250 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 设置词汇表大小、隐藏单元数和设备（如果可用则使用GPU）\n",
    "vocab_size, num_hiddens, device = len(vocab), 256, d2l.try_gpu()\n",
    "\n",
    "# 设置训练轮数和学习率\n",
    "num_epochs, lr = 500, 1\n",
    "\n",
    "# 使用只包含更新门的GRU从头开始创建一个RNN模型\n",
    "model = d2l.RNNModelScratch(len(vocab), num_hiddens, device, get_params_Z,\n",
    "                            init_gru_state, gru)\n",
    "\n",
    "# 使用提供的数据迭代器、词汇表、学习率等训练模型\n",
    "d2l.train_ch8(model, train_iter, vocab, lr, num_epochs, device)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;门控循环单元（GRU）是一种递归神经网络，具有两个门控制状态的更新和重置，这些门可以控制输入的流入和隐藏状态的保留程度。如果仅实现其中一个门，则将无法实现完整的GRU模型。  \n",
    "\n",
    "&emsp;&emsp;如果只实现了一个重置门，那么模型将只能重置隐藏状态，而无法更新其状态。这会导致模型无法学习长期依赖性并且可能无法产生准确的输出。  \n",
    "\n",
    "&emsp;&emsp;如果只实现了一个更新门，则模型可以更新其隐藏状态，但是无法选择要丢弃的信息，这也可能会导致模型无法正确地处理长期依赖性。  \n",
    "\n",
    "&emsp;&emsp;因此，为了实现GRU模型，需要同时实现更新门和重置门。这样，模型就能够更新其隐藏状态并选择要保留和丢弃的信息，从而使其能够正确地处理长期依赖性。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 9.2 长短期记忆网络（LSTM） "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.2.1\n",
    "\n",
    "调整和分析超参数对运行时间、困惑度和输出顺序的影响。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from d2l import torch as d2l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# 定义模型超参数\n",
    "# 超参数内容：[迭代周期，隐藏单元数，批量大小，小批量数据时间步数，学习率]\n",
    "hyper_0 = [500, 512, 35, 32, 1]\n",
    "hyper_1 = [250, 512, 35, 32, 1]\n",
    "hyper_2 = [500, 256, 35, 32, 1]\n",
    "hyper_3 = [500, 512, 10, 32, 1]\n",
    "hyper_4 = [500, 512, 35, 10, 1]\n",
    "hyper_5 = [500, 512, 35, 32, 0.01]\n",
    "hyper_params = [hyper_0, hyper_1, hyper_2, hyper_3, hyper_4, hyper_5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_ch8(net, train_iter, vocab, lr, num_epochs, device,\n",
    "              use_random_iter=False):\n",
    "    \"\"\"训练模型（定义见第8章）\"\"\"\n",
    "\n",
    "    # 定义交叉熵损失用于计算损失\n",
    "    loss = nn.CrossEntropyLoss()\n",
    "\n",
    "    # animator = d2l.Animator(xlabel='epoch', ylabel='perplexity',\n",
    "    #                         legend=['train'], xlim=[10, num_epochs])\n",
    "\n",
    "    # 初始化优化器（SGD）以更新模型参数\n",
    "    if isinstance(net, nn.Module):\n",
    "        updater = torch.optim.SGD(net.parameters(), lr)\n",
    "    else:\n",
    "        updater = lambda batch_size: d2l.sgd(net.params, lr, batch_size)\n",
    "\n",
    "    # 定义生成基于训练模型的预测的函数\n",
    "    predict = lambda prefix: d2l.predict_ch8(prefix, 50, net, vocab, device)\n",
    "\n",
    "    # 训练和预测循环在多个epoch上\n",
    "    for epoch in range(num_epochs):\n",
    "        # 对模型进行一个epoch的训练，并获取困惑度和速度\n",
    "        ppl, speed = d2l.train_epoch_ch8(net, train_iter, loss, updater, device, use_random_iter)\n",
    "\n",
    "        # 如果要可视化训练进度，请取消注释以下行\n",
    "        # if (epoch + 1) % 10 == 0:\n",
    "        #     animator.add(epoch + 1, [ppl])\n",
    "\n",
    "    # 打印训练后给定前缀（'time traveller'）的预测\n",
    "    print(predict('time traveller'))\n",
    "\n",
    "    # 返回训练后的困惑度\n",
    "    return ppl\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time traveller proceeded anyreal body must have extension in fou\n",
      "hyper[0]: Timer:13.66, perplexity:1.0\n",
      "time traveller dimension diftermiver for the time traveller for \n",
      "hyper[1]: Timer:8.54, perplexity:2.3\n",
      "time traveller came back andfilby s anecdote collapsedthe thing \n",
      "hyper[2]: Timer:13.14, perplexity:1.0\n",
      "time traveller smiled round at us then still smiling faintlyand \n",
      "hyper[3]: Timer:66.95, perplexity:1.0\n",
      "time traveller came back andfilby s anecdote collapsedthe thing \n",
      "hyper[4]: Timer:41.39, perplexity:1.0\n",
      "time traveller   t  t  t  t  t  t  t  t  t  t  t  t  t  t  t  t \n",
      "hyper[5]: Timer:19.46, perplexity:16.7\n"
     ]
    }
   ],
   "source": [
    "perplexity = []\n",
    "times = []\n",
    "\n",
    "# 遍历超参数列表\n",
    "for i in range(len(hyper_params)):\n",
    "    # 从超参数中获取参数值\n",
    "    num_epochs, num_hiddens, batch_size, num_steps, lr = hyper_params[i]\n",
    "    \n",
    "    # 加载数据并创建迭代器和词汇表\n",
    "    train_iter, vocab = d2l.load_data_time_machine(batch_size, num_steps)\n",
    "    \n",
    "    # 创建GRU层和RNN模型\n",
    "    gru_layer = nn.GRU(len(vocab), num_hiddens)\n",
    "    model = d2l.RNNModel(gru_layer, len(vocab))\n",
    "    \n",
    "    # 将模型移到GPU上（如果可用）\n",
    "    model = model.to(d2l.try_gpu())\n",
    "    \n",
    "    # 创建计时器并开始计时\n",
    "    timer = d2l.Timer()\n",
    "    timer.start()\n",
    "    \n",
    "    # 训练模型并获取困惑度\n",
    "    value = round(train_ch8(model, train_iter, vocab, lr, num_epochs, d2l.try_gpu()), 1)\n",
    "    \n",
    "    # 停止计时并记录时间\n",
    "    time = timer.stop()\n",
    "    \n",
    "    # 将时间和困惑度添加到相应的列表中\n",
    "    times.append(time)\n",
    "    perplexity.append(value)\n",
    "    \n",
    "    # 打印结果\n",
    "    print('hyper[%d]: Timer:%.2f, perplexity:%.1f' % (i, times[i], perplexity[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|迭代周期|隐藏单元数|批量大小|小批量数据时间步数|学习率|困惑度|运行时间|\n",
    "| ----- | ------- | ----- | --------------- | --- | ---- | ----- |\n",
    "|500|512|35|32|1|1.0|13.66|\n",
    "|250|512|35|32|1|2.3|8.54|\n",
    "|500|256|35|32|1|1.0|13.14|\n",
    "|500|512|10|32|1|1.0|66.95|\n",
    "|500|512|35|10|1|1.0|41.39|\n",
    "|500|512|35|32|0.01|16.7|19.46|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;对比hyper[0]中的默认超参数配置，我们可以发现\n",
    "- hyper[1]中缩小迭代周期，导致模型无法完全收敛，导致困惑度增大；\n",
    "- hyper[2]中减少了隐藏层单元数，使得模型在很短时间内完成收敛，并且因为数据集较为简单的原因，困惑度也降低为1；\n",
    "- hyper[3]中减小了批量大小，这使得模型的训练时间开销显著增加；\n",
    "- hyper[4]中将小批量时间步的跨度缩小，虽然也造成了时间开销的增加，但少于批量减少带来的时间增长；\n",
    "- hyper[5]中则对学习率进行了修改，可以发现过低的学习率会使得模型收敛的非常慢，造成困惑度暴涨。\n",
    "\n",
    "&emsp;&emsp;综上可以发现，迭代周期和学习率的缩小会导致模型无法在对应周期内完成收敛，导致模型训练不完全，困惑度显著增加。批量大小即时间步跨度的降低则会导致模型每次加载少量数据，变相造成时间开销的增大；而对于简单的数据，采用一个简单的模型既可以降低时间开销，也可以使得模型很好收敛，确保性能。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.2.2\n",
    "\n",
    "如何更改模型以生成适当的单词，而不是字符序列？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;将文本的生成方式由字符序列改为单词，具体可采用以下方式："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from d2l import torch as d2l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "1. 词元化处理时采用单词级别的词元化处理，使得模型学会单词间的关系而非字符间的关系。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(32775, 4580)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def load_corpus_time_machine(max_tokens=-1):\n",
    "    \"\"\"返回时光机器数据集的词元索引列表和词表\"\"\"\n",
    "    # 读取时光机器数据集的文本行\n",
    "    lines = d2l.read_time_machine()\n",
    "    \n",
    "    # 将文本行进行分词，得到词元列表\n",
    "    tokens = d2l.tokenize(lines, 'word')\n",
    "    \n",
    "    # 创建词表（Vocabulary）对象\n",
    "    vocab = d2l.Vocab(tokens)\n",
    "    \n",
    "    # 将词元列表转换为词元索引列表\n",
    "    corpus = [vocab[token] for line in tokens for token in line]\n",
    "    \n",
    "    # 如果指定了max_tokens，截取corpus的长度\n",
    "    if max_tokens > 0:\n",
    "        corpus = corpus[:max_tokens]\n",
    "    \n",
    "    # 返回词元索引列表和词表\n",
    "    return corpus, vocab\n",
    "\n",
    "# 调用load_corpus_time_machine函数加载时光机器数据集\n",
    "corpus, vocab = load_corpus_time_machine()\n",
    "\n",
    "# 输出词元索引列表和词表的长度\n",
    "len(corpus), len(vocab)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "2. 使用双向语言模型，将前后文的单词序列纳入考虑范围，实现根据上下文预测单词，获得更准确的输出。  \n",
    "&emsp;&emsp;`lstm_layer = nn.LSTM(num_inputs, num_hiddens, bidirectional=True)`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "3. 使用WordEmbedding取代One-hot，将单词映射到向量空间，保留了近义词之间的相关性，可以学习单词间的语义关系，更准确的生成单词。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "class RNNModel_Embedding(nn.Module):\n",
    "    \"\"\"使用嵌入层的RNN模型\"\"\"\n",
    "\n",
    "    def __init__(self, rnn_layer, vocab_size, embedding_dim, **kwargs):\n",
    "        super(RNNModel_Embedding, self).__init__(**kwargs)\n",
    "        self.rnn = rnn_layer\n",
    "        self.vocab_size = vocab_size\n",
    "        self.num_hiddens = self.rnn.hidden_size\n",
    "        self.embedding = nn.Embedding(vocab_size, embedding_dim)\n",
    "\n",
    "        # 如果RNN是双向的，`num_directions`应为2，否则为1\n",
    "        if not self.rnn.bidirectional:\n",
    "            self.num_directions = 1\n",
    "            self.linear = nn.Linear(self.num_hiddens, self.vocab_size)\n",
    "        else:\n",
    "            self.num_directions = 2\n",
    "            self.linear = nn.Linear(self.num_hiddens * 2, self.vocab_size)\n",
    "\n",
    "    def forward(self, inputs, state):\n",
    "        \"\"\"定义前向传播逻辑\"\"\"\n",
    "        # 将输入通过嵌入层转换为词嵌入向量\n",
    "        X = self.embedding(inputs.T.long())\n",
    "        X = X.to(torch.float32)\n",
    "        \n",
    "        # 将词嵌入向量传入RNN层，得到输出\n",
    "        Y, state = self.rnn(X, state)\n",
    "\n",
    "        # 全连接层将输出变换为形状为(`num_steps` * `batch_size`, `vocab_size`)的张量\n",
    "        output = self.linear(Y.reshape((-1, Y.shape[-1])))\n",
    "        return output, state\n",
    "\n",
    "    def begin_state(self, device, batch_size=1):\n",
    "        \"\"\"初始化RNN的隐藏状态\"\"\"\n",
    "        if not isinstance(self.rnn, nn.LSTM):\n",
    "            # `nn.GRU`接受一个张量作为隐藏状态\n",
    "            return torch.zeros((self.num_directions * self.rnn.num_layers,\n",
    "                                batch_size, self.num_hiddens),\n",
    "                               device=device)\n",
    "        else:\n",
    "            # `nn.LSTM`接受一个元组作为隐藏状态\n",
    "            return (torch.zeros((\n",
    "                self.num_directions * self.rnn.num_layers,\n",
    "                batch_size, self.num_hiddens), device=device),\n",
    "                    torch.zeros((\n",
    "                        self.num_directions * self.rnn.num_layers,\n",
    "                        batch_size, self.num_hiddens), device=device))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "class SeqDataLoader:\n",
    "    \"\"\"用于加载序列数据的迭代器\"\"\"\n",
    "    def __init__(self, batch_size, num_steps, use_random_iter, max_tokens):\n",
    "        \"\"\"初始化函数\"\"\"\n",
    "        # 根据 use_random_iter 决定使用随机迭代还是顺序迭代\n",
    "        if use_random_iter:\n",
    "            self.data_iter_fn = d2l.seq_data_iter_random\n",
    "        else:\n",
    "            self.data_iter_fn = d2l.seq_data_iter_sequential\n",
    "        # 加载时光机器数据集\n",
    "        self.corpus, self.vocab = load_corpus_time_machine(max_tokens)\n",
    "        self.batch_size, self.num_steps = batch_size, num_steps\n",
    "\n",
    "    def __iter__(self):\n",
    "        \"\"\"返回迭代器\"\"\"\n",
    "        return self.data_iter_fn(self.corpus, self.batch_size, self.num_steps)\n",
    "\n",
    "def load_data_time_machine(batch_size, num_steps,\n",
    "                           use_random_iter=False, max_tokens=10000):\n",
    "    \"\"\"返回时光机器数据集的迭代器和词汇表\"\"\"\n",
    "    # 创建 SeqDataLoader 对象\n",
    "    data_iter = SeqDataLoader(\n",
    "        batch_size, num_steps, use_random_iter, max_tokens)\n",
    "    # 返回迭代器和词汇表\n",
    "    return data_iter, data_iter.vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# 设置批量大小、时间步数和设备（如果可用则使用GPU）\n",
    "batch_size, num_steps, device = 32, 35, d2l.try_gpu()\n",
    "\n",
    "# 加载时光机器数据集的迭代器和词汇表\n",
    "train_iter, vocab = load_data_time_machine(batch_size, num_steps)\n",
    "\n",
    "# 设置输入维度（词汇表大小）、隐藏单元数和嵌入维度\n",
    "num_inputs, num_hiddens = len(vocab), 512\n",
    "embedding_dim = len(vocab)\n",
    "\n",
    "# 设置LSTM层的层数和创建LSTM层\n",
    "num_layers = 2\n",
    "lstm_layer = nn.LSTM(num_inputs, num_hiddens, num_layers, bidirectional=True)\n",
    "\n",
    "# 创建使用嵌入层的RNN模型\n",
    "model = RNNModel_Embedding(lstm_layer, len(vocab), embedding_dim=embedding_dim)\n",
    "\n",
    "# 将模型移动到指定设备（GPU或CPU）\n",
    "model = model.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def predict_word(prefix, num_preds, net, vocab, device):\n",
    "    \"\"\"根据给定的前缀生成新单词\"\"\"\n",
    "    # 初始化模型的隐藏状态\n",
    "    state = net.begin_state(batch_size=1, device=device)\n",
    "    \n",
    "    # 将前缀的第一个词加入输出列表\n",
    "    outputs = [vocab[prefix[0]]]\n",
    "    \n",
    "    # 定义一个函数用于获取模型的输入\n",
    "    get_input = lambda: d2l.reshape(d2l.tensor(\n",
    "        [outputs[-1]], device=device), (1, 1))\n",
    "    \n",
    "    # 进行预热期，通过前缀进行模型的预热\n",
    "    for y in prefix[1:]:\n",
    "        _, state = net(get_input(), state)\n",
    "        outputs.append(vocab[y])\n",
    "    \n",
    "    # 开始生成新单词\n",
    "    for _ in range(num_preds):\n",
    "        y, state = net(get_input(), state)\n",
    "        word_idx = int(y.argmax(dim=1).reshape(1))\n",
    "        word = vocab.idx_to_token[word_idx]\n",
    "        \n",
    "        # 跳过未知单词\n",
    "        if word == '<unk>':\n",
    "            continue\n",
    "        \n",
    "        # 将预测的单词加入输出列表\n",
    "        outputs.append(word_idx)\n",
    "        \n",
    "        # 如果预测的单词以空格结尾，则停止\n",
    "        if word.endswith(' '):\n",
    "            break\n",
    "    \n",
    "    # 将输出的单词索引列表转换为字符串\n",
    "    return ''.join([vocab.idx_to_token[i] for i in outputs])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def train_ch8(net, train_iter, vocab, lr, num_epochs, device,\n",
    "              use_random_iter=False):\n",
    "    \"\"\"训练模型（定义见第8章）\"\"\"\n",
    "    # 使用交叉熵损失作为损失函数\n",
    "    loss = nn.CrossEntropyLoss()\n",
    "    \n",
    "    # 创建一个动画器（animator）用于可视化训练过程\n",
    "    animator = d2l.Animator(xlabel='epoch', ylabel='perplexity',\n",
    "                            legend=['train'], xlim=[10, num_epochs])\n",
    "    \n",
    "    # 初始化优化器\n",
    "    if isinstance(net, nn.Module):\n",
    "        updater = torch.optim.SGD(net.parameters(), lr)\n",
    "    else:\n",
    "        updater = lambda batch_size: d2l.sgd(net.params, lr, batch_size)\n",
    "    \n",
    "    # 定义一个用于生成预测的函数\n",
    "    predict = lambda prefix: predict_word(prefix, 50, net, vocab, device)\n",
    "    \n",
    "    # 训练和预测\n",
    "    for epoch in range(num_epochs):\n",
    "        # 使用d2l.train_epoch_ch8进行模型训练\n",
    "        ppl, speed = d2l.train_epoch_ch8(\n",
    "            net, train_iter, loss, updater, device, use_random_iter)\n",
    "        \n",
    "        # 每隔10个周期生成一些预测结果并添加到动画中\n",
    "        if (epoch + 1) % 10 == 0:\n",
    "            print(predict('time traveller'))\n",
    "            animator.add(epoch + 1, [ppl])\n",
    "    \n",
    "    # 打印最终的困惑度和训练速度\n",
    "    print(f'perplexity {ppl:.1f}, {speed:.1f} tokens/sec on {str(device)}')\n",
    "    \n",
    "    # 打印最终的预测结果\n",
    "    print(predict('time traveller'))\n",
    "    print(predict('traveller'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "perplexity 1.0, 14810.7 tokens/sec on cuda:0\n",
      "tim<unk><unk>t<unk>av<unk><unk><unk><unk><unk>andyetbyyetbyhbyhbyhbyhbyhbyhbyhbyhbyhbyhbyhbyhbyhbyhbyhbyhbyhbyhbyhbyhbyhbyhbyh\n",
      "t<unk>av<unk><unk><unk><unk><unk>andyetbyyetbyhbyhbyhbyhbyhbyhbyhbyhbyhbyhbyhbyhbyhbyhbyhbyhbyhbyhbyhbyhbyhbyhbyh\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"259.009375pt\" height=\"183.35625pt\" viewBox=\"0 0 259.009375 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2024-02-14T19:39:23.126597</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 183.35625 \n",
       "L 259.009375 183.35625 \n",
       "L 259.009375 -0 \n",
       "L 0 -0 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 46.965625 145.8 \n",
       "L 242.265625 145.8 \n",
       "L 242.265625 7.2 \n",
       "L 46.965625 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 82.837054 145.8 \n",
       "L 82.837054 7.2 \n",
       "\" clip-path=\"url(#pbe91514b23)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"m3614ed5722\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3614ed5722\" x=\"82.837054\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(73.293304 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 122.694196 145.8 \n",
       "L 122.694196 7.2 \n",
       "\" clip-path=\"url(#pbe91514b23)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3614ed5722\" x=\"122.694196\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 200 -->\n",
       "      <g transform=\"translate(113.150446 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 162.551339 145.8 \n",
       "L 162.551339 7.2 \n",
       "\" clip-path=\"url(#pbe91514b23)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3614ed5722\" x=\"162.551339\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 300 -->\n",
       "      <g transform=\"translate(153.007589 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 202.408482 145.8 \n",
       "L 202.408482 7.2 \n",
       "\" clip-path=\"url(#pbe91514b23)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3614ed5722\" x=\"202.408482\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 400 -->\n",
       "      <g transform=\"translate(192.864732 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 242.265625 145.8 \n",
       "L 242.265625 7.2 \n",
       "\" clip-path=\"url(#pbe91514b23)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3614ed5722\" x=\"242.265625\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 500 -->\n",
       "      <g transform=\"translate(232.721875 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(129.3875 174.076563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 46.965625 140.693937 \n",
       "L 242.265625 140.693937 \n",
       "\" clip-path=\"url(#pbe91514b23)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"ma1fada56f7\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#ma1fada56f7\" x=\"46.965625\" y=\"140.693937\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(33.603125 144.493156)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 46.965625 116.964966 \n",
       "L 242.265625 116.964966 \n",
       "\" clip-path=\"url(#pbe91514b23)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#ma1fada56f7\" x=\"46.965625\" y=\"116.964966\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(27.240625 120.764185)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 46.965625 93.235995 \n",
       "L 242.265625 93.235995 \n",
       "\" clip-path=\"url(#pbe91514b23)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#ma1fada56f7\" x=\"46.965625\" y=\"93.235995\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 40 -->\n",
       "      <g transform=\"translate(27.240625 97.035214)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <path d=\"M 46.965625 69.507024 \n",
       "L 242.265625 69.507024 \n",
       "\" clip-path=\"url(#pbe91514b23)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#ma1fada56f7\" x=\"46.965625\" y=\"69.507024\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 60 -->\n",
       "      <g transform=\"translate(27.240625 73.306243)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n",
       "Q 1688 2584 1439 2293 \n",
       "Q 1191 2003 1191 1497 \n",
       "Q 1191 994 1439 701 \n",
       "Q 1688 409 2113 409 \n",
       "Q 2538 409 2786 701 \n",
       "Q 3034 994 3034 1497 \n",
       "Q 3034 2003 2786 2293 \n",
       "Q 2538 2584 2113 2584 \n",
       "z\n",
       "M 3366 4563 \n",
       "L 3366 3988 \n",
       "Q 3128 4100 2886 4159 \n",
       "Q 2644 4219 2406 4219 \n",
       "Q 1781 4219 1451 3797 \n",
       "Q 1122 3375 1075 2522 \n",
       "Q 1259 2794 1537 2939 \n",
       "Q 1816 3084 2150 3084 \n",
       "Q 2853 3084 3261 2657 \n",
       "Q 3669 2231 3669 1497 \n",
       "Q 3669 778 3244 343 \n",
       "Q 2819 -91 2113 -91 \n",
       "Q 1303 -91 875 529 \n",
       "Q 447 1150 447 2328 \n",
       "Q 447 3434 972 4092 \n",
       "Q 1497 4750 2381 4750 \n",
       "Q 2619 4750 2861 4703 \n",
       "Q 3103 4656 3366 4563 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <path d=\"M 46.965625 45.778053 \n",
       "L 242.265625 45.778053 \n",
       "\" clip-path=\"url(#pbe91514b23)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#ma1fada56f7\" x=\"46.965625\" y=\"45.778053\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 80 -->\n",
       "      <g transform=\"translate(27.240625 49.577272)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n",
       "Q 1584 2216 1326 1975 \n",
       "Q 1069 1734 1069 1313 \n",
       "Q 1069 891 1326 650 \n",
       "Q 1584 409 2034 409 \n",
       "Q 2484 409 2743 651 \n",
       "Q 3003 894 3003 1313 \n",
       "Q 3003 1734 2745 1975 \n",
       "Q 2488 2216 2034 2216 \n",
       "z\n",
       "M 1403 2484 \n",
       "Q 997 2584 770 2862 \n",
       "Q 544 3141 544 3541 \n",
       "Q 544 4100 942 4425 \n",
       "Q 1341 4750 2034 4750 \n",
       "Q 2731 4750 3128 4425 \n",
       "Q 3525 4100 3525 3541 \n",
       "Q 3525 3141 3298 2862 \n",
       "Q 3072 2584 2669 2484 \n",
       "Q 3125 2378 3379 2068 \n",
       "Q 3634 1759 3634 1313 \n",
       "Q 3634 634 3220 271 \n",
       "Q 2806 -91 2034 -91 \n",
       "Q 1263 -91 848 271 \n",
       "Q 434 634 434 1313 \n",
       "Q 434 1759 690 2068 \n",
       "Q 947 2378 1403 2484 \n",
       "z\n",
       "M 1172 3481 \n",
       "Q 1172 3119 1398 2916 \n",
       "Q 1625 2713 2034 2713 \n",
       "Q 2441 2713 2670 2916 \n",
       "Q 2900 3119 2900 3481 \n",
       "Q 2900 3844 2670 4047 \n",
       "Q 2441 4250 2034 4250 \n",
       "Q 1625 4250 1398 4047 \n",
       "Q 1172 3844 1172 3481 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-38\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_21\">\n",
       "      <path d=\"M 46.965625 22.049083 \n",
       "L 242.265625 22.049083 \n",
       "\" clip-path=\"url(#pbe91514b23)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_22\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#ma1fada56f7\" x=\"46.965625\" y=\"22.049083\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(20.878125 25.848301)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- perplexity -->\n",
       "     <g transform=\"translate(14.798437 101.626563)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"229.589844\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"257.373047\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"317.146484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"376.326172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"404.109375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"443.318359\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_23\">\n",
       "    <path d=\"M 46.965625 13.5 \n",
       "L 50.951339 117.19174 \n",
       "L 54.937054 133.769478 \n",
       "L 58.922768 137.845377 \n",
       "L 62.908482 138.83417 \n",
       "L 66.894196 139.158689 \n",
       "L 70.879911 139.322166 \n",
       "L 74.865625 139.27366 \n",
       "L 78.851339 139.282629 \n",
       "L 82.837054 139.388285 \n",
       "L 86.822768 139.396683 \n",
       "L 90.808482 139.366554 \n",
       "L 94.794196 139.410135 \n",
       "L 98.779911 139.448032 \n",
       "L 102.765625 139.449771 \n",
       "L 106.751339 139.432963 \n",
       "L 110.737054 139.45623 \n",
       "L 114.722768 139.464247 \n",
       "L 118.708482 139.441468 \n",
       "L 122.694196 139.461934 \n",
       "L 126.679911 139.45095 \n",
       "L 130.665625 139.469644 \n",
       "L 134.651339 139.482941 \n",
       "L 138.637054 139.474599 \n",
       "L 142.622768 139.480258 \n",
       "L 146.608482 139.484763 \n",
       "L 150.594196 139.47396 \n",
       "L 154.579911 139.488743 \n",
       "L 158.565625 139.4876 \n",
       "L 162.551339 139.4861 \n",
       "L 166.537054 139.487073 \n",
       "L 170.522768 139.492587 \n",
       "L 174.508482 139.494019 \n",
       "L 178.494196 139.48531 \n",
       "L 182.479911 139.494825 \n",
       "L 186.465625 139.473771 \n",
       "L 190.451339 139.49103 \n",
       "L 194.437054 139.493591 \n",
       "L 198.422768 139.497334 \n",
       "L 202.408482 139.495977 \n",
       "L 206.394196 139.496995 \n",
       "L 210.379911 139.488588 \n",
       "L 214.365625 139.496932 \n",
       "L 218.351339 139.497521 \n",
       "L 222.337054 139.497168 \n",
       "L 226.322768 139.49932 \n",
       "L 230.308482 139.498813 \n",
       "L 234.294196 139.497771 \n",
       "L 238.279911 139.49349 \n",
       "L 242.265625 139.5 \n",
       "\" clip-path=\"url(#pbe91514b23)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 46.965625 145.8 \n",
       "L 46.965625 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 242.265625 145.8 \n",
       "L 242.265625 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 46.965625 145.8 \n",
       "L 242.265625 145.8 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 46.965625 7.2 \n",
       "L 242.265625 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 179.990625 29.878125 \n",
       "L 235.265625 29.878125 \n",
       "Q 237.265625 29.878125 237.265625 27.878125 \n",
       "L 237.265625 14.2 \n",
       "Q 237.265625 12.2 235.265625 12.2 \n",
       "L 179.990625 12.2 \n",
       "Q 177.990625 12.2 177.990625 14.2 \n",
       "L 177.990625 27.878125 \n",
       "Q 177.990625 29.878125 179.990625 29.878125 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_24\">\n",
       "     <path d=\"M 181.990625 20.298438 \n",
       "L 191.990625 20.298438 \n",
       "L 201.990625 20.298438 \n",
       "\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    </g>\n",
       "    <g id=\"text_14\">\n",
       "     <!-- train -->\n",
       "     <g transform=\"translate(209.990625 23.798438)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pbe91514b23\">\n",
       "   <rect x=\"46.965625\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 350x250 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 设置训练的超参数：训练周期数和学习率\n",
    "num_epochs, lr = 500, 1\n",
    "\n",
    "# 调用train_ch8函数进行模型训练\n",
    "train_ch8(model, train_iter, vocab, lr, num_epochs, device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;在相同的超参数训练下，使用单词会比使用字符串更快地拟合，从非常高的困惑值在相较于使用字符串更短的迭代次数下降到 `1.0` 的困惑值。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.2.3\n",
    "\n",
    "在给定隐藏层维度的情况下，比较门控循环单元、长短期记忆网络和常规循环神经网络的计算成本。要特别注意训练和推断成本。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;将隐藏层维度统一设置为256"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# [迭代周期，隐藏单元数，批量大小，小批量数据时间步数，学习率]\n",
    "num_epochs, num_hiddens, batch_size, num_steps, lr = [500, 256, 35, 32, 1]\n",
    "train_iter, vocab = d2l.load_data_time_machine(batch_size, num_steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_ch8(net, train_iter, vocab, lr, num_epochs, device,\n",
    "              use_random_iter=False):\n",
    "    \"\"\"\n",
    "    训练模型（定义见第8章）\n",
    "    \"\"\"\n",
    "    # 交叉熵损失函数\n",
    "    loss = nn.CrossEntropyLoss()\n",
    "\n",
    "    # 初始化优化器\n",
    "    if isinstance(net, nn.Module):\n",
    "        updater = torch.optim.SGD(net.parameters(), lr)\n",
    "    else:\n",
    "        updater = lambda batch_size: d2l.sgd(net.params, lr, batch_size)\n",
    "\n",
    "    # 预测函数\n",
    "    predict = lambda prefix: d2l.predict_ch8(prefix, 50, net, vocab, device)\n",
    "\n",
    "    # 训练和预测\n",
    "    for epoch in range(num_epochs):\n",
    "        # 调用train_epoch_ch8函数进行一轮训练\n",
    "        ppl, speed = d2l.train_epoch_ch8(net, train_iter, loss, updater, device, use_random_iter)\n",
    "        \n",
    "        # 每10轮打印一次困惑度\n",
    "        if (epoch + 1) % 10 == 0:\n",
    "            print(f'Epoch {epoch + 1}, Perplexity: {ppl:.1f}')\n",
    "\n",
    "    # 打印一个例子的预测结果\n",
    "    print(predict('time traveller'))\n",
    "\n",
    "    # 返回最终的困惑度值\n",
    "    return ppl\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "perplexity 1.3, 358070.8 tokens/sec on cuda:0\n",
      "time traveller smiled round at us then still smilissiepare haing\n",
      "traveller carexdineculencas ingrame trout maineplyou difent\n",
      "Timer: 18.04\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"252.646875pt\" height=\"183.35625pt\" viewBox=\"0 0 252.646875 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2024-02-14T19:39:41.361752</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 183.35625 \n",
       "L 252.646875 183.35625 \n",
       "L 252.646875 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "L 40.603125 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 76.474554 145.8 \n",
       "L 76.474554 7.2 \n",
       "\" clip-path=\"url(#p462e5cd396)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"mb7b89f57f6\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mb7b89f57f6\" x=\"76.474554\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(66.930804 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 116.331696 145.8 \n",
       "L 116.331696 7.2 \n",
       "\" clip-path=\"url(#p462e5cd396)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mb7b89f57f6\" x=\"116.331696\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 200 -->\n",
       "      <g transform=\"translate(106.787946 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 156.188839 145.8 \n",
       "L 156.188839 7.2 \n",
       "\" clip-path=\"url(#p462e5cd396)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mb7b89f57f6\" x=\"156.188839\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 300 -->\n",
       "      <g transform=\"translate(146.645089 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 196.045982 145.8 \n",
       "L 196.045982 7.2 \n",
       "\" clip-path=\"url(#p462e5cd396)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mb7b89f57f6\" x=\"196.045982\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 400 -->\n",
       "      <g transform=\"translate(186.502232 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "\" clip-path=\"url(#p462e5cd396)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mb7b89f57f6\" x=\"235.903125\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 500 -->\n",
       "      <g transform=\"translate(226.359375 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(123.025 174.076563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 40.603125 130.196127 \n",
       "L 235.903125 130.196127 \n",
       "\" clip-path=\"url(#p462e5cd396)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"m46406754d7\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m46406754d7\" x=\"40.603125\" y=\"130.196127\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(27.240625 133.995346)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 40.603125 104.848572 \n",
       "L 235.903125 104.848572 \n",
       "\" clip-path=\"url(#p462e5cd396)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m46406754d7\" x=\"40.603125\" y=\"104.848572\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 4 -->\n",
       "      <g transform=\"translate(27.240625 108.64779)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 40.603125 79.501016 \n",
       "L 235.903125 79.501016 \n",
       "\" clip-path=\"url(#p462e5cd396)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m46406754d7\" x=\"40.603125\" y=\"79.501016\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 6 -->\n",
       "      <g transform=\"translate(27.240625 83.300235)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n",
       "Q 1688 2584 1439 2293 \n",
       "Q 1191 2003 1191 1497 \n",
       "Q 1191 994 1439 701 \n",
       "Q 1688 409 2113 409 \n",
       "Q 2538 409 2786 701 \n",
       "Q 3034 994 3034 1497 \n",
       "Q 3034 2003 2786 2293 \n",
       "Q 2538 2584 2113 2584 \n",
       "z\n",
       "M 3366 4563 \n",
       "L 3366 3988 \n",
       "Q 3128 4100 2886 4159 \n",
       "Q 2644 4219 2406 4219 \n",
       "Q 1781 4219 1451 3797 \n",
       "Q 1122 3375 1075 2522 \n",
       "Q 1259 2794 1537 2939 \n",
       "Q 1816 3084 2150 3084 \n",
       "Q 2853 3084 3261 2657 \n",
       "Q 3669 2231 3669 1497 \n",
       "Q 3669 778 3244 343 \n",
       "Q 2819 -91 2113 -91 \n",
       "Q 1303 -91 875 529 \n",
       "Q 447 1150 447 2328 \n",
       "Q 447 3434 972 4092 \n",
       "Q 1497 4750 2381 4750 \n",
       "Q 2619 4750 2861 4703 \n",
       "Q 3103 4656 3366 4563 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <path d=\"M 40.603125 54.153461 \n",
       "L 235.903125 54.153461 \n",
       "\" clip-path=\"url(#p462e5cd396)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m46406754d7\" x=\"40.603125\" y=\"54.153461\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 8 -->\n",
       "      <g transform=\"translate(27.240625 57.952679)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n",
       "Q 1584 2216 1326 1975 \n",
       "Q 1069 1734 1069 1313 \n",
       "Q 1069 891 1326 650 \n",
       "Q 1584 409 2034 409 \n",
       "Q 2484 409 2743 651 \n",
       "Q 3003 894 3003 1313 \n",
       "Q 3003 1734 2745 1975 \n",
       "Q 2488 2216 2034 2216 \n",
       "z\n",
       "M 1403 2484 \n",
       "Q 997 2584 770 2862 \n",
       "Q 544 3141 544 3541 \n",
       "Q 544 4100 942 4425 \n",
       "Q 1341 4750 2034 4750 \n",
       "Q 2731 4750 3128 4425 \n",
       "Q 3525 4100 3525 3541 \n",
       "Q 3525 3141 3298 2862 \n",
       "Q 3072 2584 2669 2484 \n",
       "Q 3125 2378 3379 2068 \n",
       "Q 3634 1759 3634 1313 \n",
       "Q 3634 634 3220 271 \n",
       "Q 2806 -91 2034 -91 \n",
       "Q 1263 -91 848 271 \n",
       "Q 434 634 434 1313 \n",
       "Q 434 1759 690 2068 \n",
       "Q 947 2378 1403 2484 \n",
       "z\n",
       "M 1172 3481 \n",
       "Q 1172 3119 1398 2916 \n",
       "Q 1625 2713 2034 2713 \n",
       "Q 2441 2713 2670 2916 \n",
       "Q 2900 3119 2900 3481 \n",
       "Q 2900 3844 2670 4047 \n",
       "Q 2441 4250 2034 4250 \n",
       "Q 1625 4250 1398 4047 \n",
       "Q 1172 3844 1172 3481 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-38\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <path d=\"M 40.603125 28.805905 \n",
       "L 235.903125 28.805905 \n",
       "\" clip-path=\"url(#p462e5cd396)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m46406754d7\" x=\"40.603125\" y=\"28.805905\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(20.878125 32.605124)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_12\">\n",
       "     <!-- perplexity -->\n",
       "     <g transform=\"translate(14.798437 101.626563)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"229.589844\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"257.373047\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"317.146484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"376.326172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"404.109375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"443.318359\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_21\">\n",
       "    <path d=\"M 40.603125 13.5 \n",
       "L 44.588839 38.661535 \n",
       "L 48.574554 45.373428 \n",
       "L 52.560268 56.365622 \n",
       "L 56.545982 61.183512 \n",
       "L 60.531696 68.954039 \n",
       "L 64.517411 76.103249 \n",
       "L 68.503125 86.957721 \n",
       "L 72.488839 98.205646 \n",
       "L 76.474554 109.031924 \n",
       "L 80.460268 115.899258 \n",
       "L 84.445982 120.964537 \n",
       "L 88.431696 123.396814 \n",
       "L 92.417411 128.050524 \n",
       "L 96.403125 128.434626 \n",
       "L 100.388839 130.243037 \n",
       "L 104.374554 132.496371 \n",
       "L 108.360268 131.923436 \n",
       "L 112.345982 134.40693 \n",
       "L 116.331696 135.063111 \n",
       "L 120.317411 135.953376 \n",
       "L 124.303125 135.056554 \n",
       "L 128.288839 136.29337 \n",
       "L 132.274554 137.309133 \n",
       "L 136.260268 136.567114 \n",
       "L 140.245982 136.78196 \n",
       "L 144.231696 137.505419 \n",
       "L 148.217411 137.447283 \n",
       "L 152.203125 137.695525 \n",
       "L 156.188839 138.039804 \n",
       "L 160.174554 138.117733 \n",
       "L 164.160268 138.169576 \n",
       "L 168.145982 137.807058 \n",
       "L 172.131696 137.905784 \n",
       "L 176.117411 138.325544 \n",
       "L 180.103125 138.332249 \n",
       "L 184.088839 138.870716 \n",
       "L 188.074554 138.691752 \n",
       "L 192.060268 138.541337 \n",
       "L 196.045982 137.650532 \n",
       "L 200.031696 138.769326 \n",
       "L 204.017411 138.622777 \n",
       "L 208.003125 138.625408 \n",
       "L 211.988839 138.769427 \n",
       "L 215.974554 138.704266 \n",
       "L 219.960268 138.768897 \n",
       "L 223.945982 138.436716 \n",
       "L 227.931696 139.5 \n",
       "L 231.917411 137.848152 \n",
       "L 235.903125 139.193695 \n",
       "\" clip-path=\"url(#p462e5cd396)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 40.603125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 235.903125 145.8 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 40.603125 7.2 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 173.628125 29.878125 \n",
       "L 228.903125 29.878125 \n",
       "Q 230.903125 29.878125 230.903125 27.878125 \n",
       "L 230.903125 14.2 \n",
       "Q 230.903125 12.2 228.903125 12.2 \n",
       "L 173.628125 12.2 \n",
       "Q 171.628125 12.2 171.628125 14.2 \n",
       "L 171.628125 27.878125 \n",
       "Q 171.628125 29.878125 173.628125 29.878125 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_22\">\n",
       "     <path d=\"M 175.628125 20.298438 \n",
       "L 185.628125 20.298438 \n",
       "L 195.628125 20.298438 \n",
       "\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- train -->\n",
       "     <g transform=\"translate(203.628125 23.798438)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p462e5cd396\">\n",
       "   <rect x=\"40.603125\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 350x250 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#RNN\n",
    "import torch.nn as nn\n",
    "import d2l.torch as d2l\n",
    "\n",
    "# 使用指定的词汇表大小和隐藏单元数量定义一个RNN层\n",
    "rnn_layer = nn.RNN(len(vocab), num_hiddens)\n",
    "\n",
    "# 使用定义好的RNN层创建一个RNN模型\n",
    "rnn = d2l.RNNModel(rnn_layer, len(vocab))\n",
    "\n",
    "# 如果GPU可用，将模型移动到GPU上\n",
    "rnn = rnn.to(d2l.try_gpu())\n",
    "\n",
    "# 初始化一个计时器来跟踪训练时间\n",
    "timer = d2l.Timer()\n",
    "timer.start()\n",
    "\n",
    "# 使用训练迭代器、词汇表、学习率和训练轮数训练RNN模型\n",
    "d2l.train_ch8(rnn, train_iter, vocab, lr, num_epochs, d2l.try_gpu())\n",
    "\n",
    "# 停止计时器并打印经过的时间\n",
    "time = timer.stop()\n",
    "print('Timer: %.2f' % time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "perplexity 1.0, 281955.2 tokens/sec on cuda:0\n",
      "time traveller came back andfilby s anecdote collapsedthe thing \n",
      "traveller proceeded anyreal body must have extension in fou\n",
      "Timer: 23.38\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"252.646875pt\" height=\"183.35625pt\" viewBox=\"0 0 252.646875 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2024-02-14T19:40:04.810173</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 183.35625 \n",
       "L 252.646875 183.35625 \n",
       "L 252.646875 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "L 40.603125 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 76.474554 145.8 \n",
       "L 76.474554 7.2 \n",
       "\" clip-path=\"url(#p63654bfc14)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"mc544bd348a\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc544bd348a\" x=\"76.474554\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(66.930804 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 116.331696 145.8 \n",
       "L 116.331696 7.2 \n",
       "\" clip-path=\"url(#p63654bfc14)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc544bd348a\" x=\"116.331696\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 200 -->\n",
       "      <g transform=\"translate(106.787946 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 156.188839 145.8 \n",
       "L 156.188839 7.2 \n",
       "\" clip-path=\"url(#p63654bfc14)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc544bd348a\" x=\"156.188839\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 300 -->\n",
       "      <g transform=\"translate(146.645089 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 196.045982 145.8 \n",
       "L 196.045982 7.2 \n",
       "\" clip-path=\"url(#p63654bfc14)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc544bd348a\" x=\"196.045982\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 400 -->\n",
       "      <g transform=\"translate(186.502232 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "\" clip-path=\"url(#p63654bfc14)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc544bd348a\" x=\"235.903125\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 500 -->\n",
       "      <g transform=\"translate(226.359375 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(123.025 174.076563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 40.603125 105.223759 \n",
       "L 235.903125 105.223759 \n",
       "\" clip-path=\"url(#p63654bfc14)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"m3c0c076e28\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3c0c076e28\" x=\"40.603125\" y=\"105.223759\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(27.240625 109.022977)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 40.603125 61.977306 \n",
       "L 235.903125 61.977306 \n",
       "\" clip-path=\"url(#p63654bfc14)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3c0c076e28\" x=\"40.603125\" y=\"61.977306\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(20.878125 65.776525)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 40.603125 18.730853 \n",
       "L 235.903125 18.730853 \n",
       "\" clip-path=\"url(#p63654bfc14)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3c0c076e28\" x=\"40.603125\" y=\"18.730853\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 15 -->\n",
       "      <g transform=\"translate(20.878125 22.530072)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_10\">\n",
       "     <!-- perplexity -->\n",
       "     <g transform=\"translate(14.798437 101.626563)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"229.589844\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"257.373047\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"317.146484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"376.326172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"404.109375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"443.318359\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_17\">\n",
       "    <path d=\"M 40.603125 13.5 \n",
       "L 44.588839 43.8382 \n",
       "L 48.574554 55.324984 \n",
       "L 52.560268 61.973592 \n",
       "L 56.545982 66.319024 \n",
       "L 60.531696 67.714507 \n",
       "L 64.517411 72.250203 \n",
       "L 68.503125 73.490872 \n",
       "L 72.488839 78.525707 \n",
       "L 76.474554 81.522861 \n",
       "L 80.460268 83.276402 \n",
       "L 84.445982 86.756723 \n",
       "L 88.431696 90.571155 \n",
       "L 92.417411 93.093533 \n",
       "L 96.403125 96.231528 \n",
       "L 100.388839 99.818681 \n",
       "L 104.374554 103.200919 \n",
       "L 108.360268 106.733953 \n",
       "L 112.345982 111.501825 \n",
       "L 116.331696 115.928065 \n",
       "L 120.317411 119.202049 \n",
       "L 124.303125 123.146419 \n",
       "L 128.288839 127.28451 \n",
       "L 132.274554 131.428972 \n",
       "L 136.260268 133.805339 \n",
       "L 140.245982 135.474821 \n",
       "L 144.231696 137.189496 \n",
       "L 148.217411 138.086981 \n",
       "L 152.203125 138.524519 \n",
       "L 156.188839 138.435256 \n",
       "L 160.174554 138.859384 \n",
       "L 164.160268 138.848815 \n",
       "L 168.145982 139.063086 \n",
       "L 172.131696 139.027917 \n",
       "L 176.117411 139.105997 \n",
       "L 180.103125 139.174943 \n",
       "L 184.088839 138.78923 \n",
       "L 188.074554 139.291963 \n",
       "L 192.060268 139.360753 \n",
       "L 196.045982 139.317683 \n",
       "L 200.031696 139.330463 \n",
       "L 204.017411 139.284042 \n",
       "L 208.003125 139.432563 \n",
       "L 211.988839 139.372567 \n",
       "L 215.974554 139.5 \n",
       "L 219.960268 139.451416 \n",
       "L 223.945982 139.428898 \n",
       "L 227.931696 139.463434 \n",
       "L 231.917411 139.427665 \n",
       "L 235.903125 139.460133 \n",
       "\" clip-path=\"url(#p63654bfc14)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 40.603125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 235.903125 145.8 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 40.603125 7.2 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 173.628125 29.878125 \n",
       "L 228.903125 29.878125 \n",
       "Q 230.903125 29.878125 230.903125 27.878125 \n",
       "L 230.903125 14.2 \n",
       "Q 230.903125 12.2 228.903125 12.2 \n",
       "L 173.628125 12.2 \n",
       "Q 171.628125 12.2 171.628125 14.2 \n",
       "L 171.628125 27.878125 \n",
       "Q 171.628125 29.878125 173.628125 29.878125 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_18\">\n",
       "     <path d=\"M 175.628125 20.298438 \n",
       "L 185.628125 20.298438 \n",
       "L 195.628125 20.298438 \n",
       "\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    </g>\n",
       "    <g id=\"text_11\">\n",
       "     <!-- train -->\n",
       "     <g transform=\"translate(203.628125 23.798438)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p63654bfc14\">\n",
       "   <rect x=\"40.603125\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 350x250 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# GRU\n",
    "import torch.nn as nn\n",
    "import d2l.torch as d2l\n",
    "\n",
    "# 使用指定的词汇表大小和隐藏单元数量定义一个GRU层\n",
    "gru_layer = nn.GRU(len(vocab), num_hiddens)\n",
    "\n",
    "# 使用定义好的GRU层创建一个GRU模型\n",
    "gru = d2l.RNNModel(gru_layer, len(vocab))\n",
    "\n",
    "# 如果GPU可用，将模型移动到GPU上\n",
    "gru = gru.to(d2l.try_gpu())\n",
    "\n",
    "# 初始化一个计时器来跟踪训练时间\n",
    "timer = d2l.Timer()\n",
    "timer.start()\n",
    "\n",
    "# 使用训练迭代器、词汇表、学习率和训练轮数训练GRU模型\n",
    "d2l.train_ch8(gru, train_iter, vocab, lr, num_epochs, d2l.try_gpu())\n",
    "\n",
    "# 停止计时器并打印经过的时间\n",
    "time = timer.stop()\n",
    "print('Timer: %.2f' % time)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "perplexity 1.1, 229065.6 tokens/sec on cuda:0\n",
      "time traveller came back andfilby s anecdote collapsedthe thing \n",
      "traveller came back andfilby s anecdote collapsedthe thing \n",
      "Timer: 24.11\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"252.646875pt\" height=\"183.35625pt\" viewBox=\"0 0 252.646875 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2024-02-14T19:40:29.002359</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 183.35625 \n",
       "L 252.646875 183.35625 \n",
       "L 252.646875 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "L 40.603125 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 76.474554 145.8 \n",
       "L 76.474554 7.2 \n",
       "\" clip-path=\"url(#pc5362b166a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"m525a0e8c6a\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m525a0e8c6a\" x=\"76.474554\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(66.930804 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 116.331696 145.8 \n",
       "L 116.331696 7.2 \n",
       "\" clip-path=\"url(#pc5362b166a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m525a0e8c6a\" x=\"116.331696\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 200 -->\n",
       "      <g transform=\"translate(106.787946 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 156.188839 145.8 \n",
       "L 156.188839 7.2 \n",
       "\" clip-path=\"url(#pc5362b166a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m525a0e8c6a\" x=\"156.188839\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 300 -->\n",
       "      <g transform=\"translate(146.645089 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 196.045982 145.8 \n",
       "L 196.045982 7.2 \n",
       "\" clip-path=\"url(#pc5362b166a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m525a0e8c6a\" x=\"196.045982\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 400 -->\n",
       "      <g transform=\"translate(186.502232 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "\" clip-path=\"url(#pc5362b166a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m525a0e8c6a\" x=\"235.903125\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 500 -->\n",
       "      <g transform=\"translate(226.359375 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(123.025 174.076563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 40.603125 108.617441 \n",
       "L 235.903125 108.617441 \n",
       "\" clip-path=\"url(#pc5362b166a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"m05faf7517e\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m05faf7517e\" x=\"40.603125\" y=\"108.617441\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(27.240625 112.416659)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 40.603125 69.559726 \n",
       "L 235.903125 69.559726 \n",
       "\" clip-path=\"url(#pc5362b166a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m05faf7517e\" x=\"40.603125\" y=\"69.559726\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(20.878125 73.358945)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 40.603125 30.502011 \n",
       "L 235.903125 30.502011 \n",
       "\" clip-path=\"url(#pc5362b166a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m05faf7517e\" x=\"40.603125\" y=\"30.502011\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 15 -->\n",
       "      <g transform=\"translate(20.878125 34.30123)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_10\">\n",
       "     <!-- perplexity -->\n",
       "     <g transform=\"translate(14.798437 101.626563)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"229.589844\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"257.373047\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"317.146484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"376.326172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"404.109375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"443.318359\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_17\">\n",
       "    <path d=\"M 40.603125 13.5 \n",
       "L 44.588839 21.508987 \n",
       "L 48.574554 35.470836 \n",
       "L 52.560268 51.346779 \n",
       "L 56.545982 61.865366 \n",
       "L 60.531696 66.384214 \n",
       "L 64.517411 69.533144 \n",
       "L 68.503125 72.882771 \n",
       "L 72.488839 75.784247 \n",
       "L 76.474554 77.766068 \n",
       "L 80.460268 82.534664 \n",
       "L 84.445982 86.663994 \n",
       "L 88.431696 89.023758 \n",
       "L 92.417411 92.955441 \n",
       "L 96.403125 96.604804 \n",
       "L 100.388839 99.328613 \n",
       "L 104.374554 104.270641 \n",
       "L 108.360268 106.613149 \n",
       "L 112.345982 110.343834 \n",
       "L 116.331696 113.95094 \n",
       "L 120.317411 117.59852 \n",
       "L 124.303125 121.125109 \n",
       "L 128.288839 123.931485 \n",
       "L 132.274554 126.379886 \n",
       "L 136.260268 130.361762 \n",
       "L 140.245982 131.559123 \n",
       "L 144.231696 133.628825 \n",
       "L 148.217411 135.212631 \n",
       "L 152.203125 136.185354 \n",
       "L 156.188839 137.529236 \n",
       "L 160.174554 137.798853 \n",
       "L 164.160268 137.700984 \n",
       "L 168.145982 138.75147 \n",
       "L 172.131696 138.142227 \n",
       "L 176.117411 138.782667 \n",
       "L 180.103125 139.21043 \n",
       "L 184.088839 139.344264 \n",
       "L 188.074554 139.342579 \n",
       "L 192.060268 138.830001 \n",
       "L 196.045982 139.234402 \n",
       "L 200.031696 139.210838 \n",
       "L 204.017411 139.351831 \n",
       "L 208.003125 139.375791 \n",
       "L 211.988839 139.5 \n",
       "L 215.974554 139.47866 \n",
       "L 219.960268 139.469916 \n",
       "L 223.945982 139.376791 \n",
       "L 227.931696 139.489025 \n",
       "L 231.917411 139.388926 \n",
       "L 235.903125 139.436524 \n",
       "\" clip-path=\"url(#pc5362b166a)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 40.603125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 235.903125 145.8 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 40.603125 7.2 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 173.628125 29.878125 \n",
       "L 228.903125 29.878125 \n",
       "Q 230.903125 29.878125 230.903125 27.878125 \n",
       "L 230.903125 14.2 \n",
       "Q 230.903125 12.2 228.903125 12.2 \n",
       "L 173.628125 12.2 \n",
       "Q 171.628125 12.2 171.628125 14.2 \n",
       "L 171.628125 27.878125 \n",
       "Q 171.628125 29.878125 173.628125 29.878125 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_18\">\n",
       "     <path d=\"M 175.628125 20.298438 \n",
       "L 185.628125 20.298438 \n",
       "L 195.628125 20.298438 \n",
       "\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    </g>\n",
       "    <g id=\"text_11\">\n",
       "     <!-- train -->\n",
       "     <g transform=\"translate(203.628125 23.798438)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pc5362b166a\">\n",
       "   <rect x=\"40.603125\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 350x250 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# LSTM\n",
    "import torch.nn as nn\n",
    "import d2l.torch as d2l\n",
    "\n",
    "# 使用指定的词汇表大小和隐藏单元数量定义一个LSTM层\n",
    "lstm_layer = nn.LSTM(len(vocab), num_hiddens)\n",
    "\n",
    "# 使用定义好的LSTM层创建一个LSTM模型\n",
    "lstm = d2l.RNNModel(lstm_layer, len(vocab))\n",
    "\n",
    "# 如果GPU可用，将模型移动到GPU上\n",
    "lstm = lstm.to(d2l.try_gpu())\n",
    "\n",
    "# 初始化一个计时器来跟踪训练时间\n",
    "timer = d2l.Timer()\n",
    "timer.start()\n",
    "\n",
    "# 使用训练迭代器、词汇表、学习率和训练轮数训练LSTM模型\n",
    "d2l.train_ch8(lstm, train_iter, vocab, lr, num_epochs, d2l.try_gpu())\n",
    "\n",
    "# 停止计时器并打印经过的时间\n",
    "time = timer.stop()\n",
    "print('Timer: %.2f' % time)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| 模型 | 困惑度 | 运行时间 |\n",
    "| ---- | ----- | ---- |\n",
    "|RNN|1.3|18.04|\n",
    "|GRU|1.0|23.38|\n",
    "|LSTM|1.1|24.11|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;可以从上述表中看出结构最简单的RNN所需计算成本最低，运行时间最短。GRU在RNN基础上增加了更新门和重置门两个门控单元，结构复杂，参数量也有所增加，所需的计算成本要大于RNN，运行时间也更长，但困惑度有随之降低，模型准确度更高。而LSTM内含输入门、输出门以及遗忘门3个门，结构进一步复杂化，计算成本也是三个模型中最高的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.2.4  \n",
    "\n",
    "既然候选记忆元通过使用 $\\tanh$ 函数来确保值范围在 $(-1,1)$ 之间，那么为什么隐状态需要再次使用 $\\tanh$ 函数来确保输出值范围在 $(-1,1)$ 之间呢？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;在整个LSTM计算中，候选记忆元 $\\tilde{C}_t$ 的计算公式 $9.2.2$：\n",
    "$$\\tilde{C}_t=\\tanh(X_tW_{xc}+H_{t-1}W_{hc}+b_c)$$\n",
    "&emsp;&emsp;此时 $\\tanh$ 为激活函数，确保 $\\tilde{C}_t$ 的值域在 $(-1,1)$ 之间。  \n",
    "\n",
    "&emsp;&emsp;而在计算记忆元 $C_t$ 时，存在输入门和遗忘门对记忆元的值进行影响，具体计算公式 $9.2.3$ 如下：\n",
    "$$C_t=F_t \\odot C_{t-1}+I_t \\odot \\tilde{C}_t$$\n",
    "&emsp;&emsp;上述公式中，输入门和遗忘门的值域为 $(0,1)$ ，因此当 $F_t$、$I_t$ 的值均为 $1$ ，且 $C_{t-1}、\\tilde{C}_t$ 的值也均为 $1$ 时，整个记忆元的最终输出结果为 $2$ 。此时对于整个隐状态而言，由于输出门的值域也为 $(0,1)$ ，因此隐状态值很有可能超过 $1$ ，因此需要对记忆元通过 $\\tanh$ 函数来进行约束取值范围，以此确保输出值范围在 $(-1,1)$ 之间。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.2.5\n",
    "\n",
    "实现一个能够基于时间序列进行预测而不是基于字符序列进行预测的长短期记忆网络模型。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;这里使用[8.1.序列模型](https://zh.d2l.ai/chapter_recurrent-neural-networks/sequence.html#id11)中的序列数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"406.885938pt\" height=\"211.07625pt\" viewBox=\"0 0 406.885938 211.07625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2024-02-14T19:40:29.129882</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 211.07625 \n",
       "L 406.885938 211.07625 \n",
       "L 406.885938 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 52.160938 173.52 \n",
       "L 386.960938 173.52 \n",
       "L 386.960938 7.2 \n",
       "L 52.160938 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 118.852829 173.52 \n",
       "L 118.852829 7.2 \n",
       "\" clip-path=\"url(#p6908287b77)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"m34046a9be5\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m34046a9be5\" x=\"118.852829\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 200 -->\n",
       "      <g transform=\"translate(109.309079 188.118438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 185.879856 173.52 \n",
       "L 185.879856 7.2 \n",
       "\" clip-path=\"url(#p6908287b77)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m34046a9be5\" x=\"185.879856\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 400 -->\n",
       "      <g transform=\"translate(176.336106 188.118438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 252.906883 173.52 \n",
       "L 252.906883 7.2 \n",
       "\" clip-path=\"url(#p6908287b77)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m34046a9be5\" x=\"252.906883\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 600 -->\n",
       "      <g transform=\"translate(243.363133 188.118438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n",
       "Q 1688 2584 1439 2293 \n",
       "Q 1191 2003 1191 1497 \n",
       "Q 1191 994 1439 701 \n",
       "Q 1688 409 2113 409 \n",
       "Q 2538 409 2786 701 \n",
       "Q 3034 994 3034 1497 \n",
       "Q 3034 2003 2786 2293 \n",
       "Q 2538 2584 2113 2584 \n",
       "z\n",
       "M 3366 4563 \n",
       "L 3366 3988 \n",
       "Q 3128 4100 2886 4159 \n",
       "Q 2644 4219 2406 4219 \n",
       "Q 1781 4219 1451 3797 \n",
       "Q 1122 3375 1075 2522 \n",
       "Q 1259 2794 1537 2939 \n",
       "Q 1816 3084 2150 3084 \n",
       "Q 2853 3084 3261 2657 \n",
       "Q 3669 2231 3669 1497 \n",
       "Q 3669 778 3244 343 \n",
       "Q 2819 -91 2113 -91 \n",
       "Q 1303 -91 875 529 \n",
       "Q 447 1150 447 2328 \n",
       "Q 447 3434 972 4092 \n",
       "Q 1497 4750 2381 4750 \n",
       "Q 2619 4750 2861 4703 \n",
       "Q 3103 4656 3366 4563 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 319.93391 173.52 \n",
       "L 319.93391 7.2 \n",
       "\" clip-path=\"url(#p6908287b77)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m34046a9be5\" x=\"319.93391\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 800 -->\n",
       "      <g transform=\"translate(310.39016 188.118438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n",
       "Q 1584 2216 1326 1975 \n",
       "Q 1069 1734 1069 1313 \n",
       "Q 1069 891 1326 650 \n",
       "Q 1584 409 2034 409 \n",
       "Q 2484 409 2743 651 \n",
       "Q 3003 894 3003 1313 \n",
       "Q 3003 1734 2745 1975 \n",
       "Q 2488 2216 2034 2216 \n",
       "z\n",
       "M 1403 2484 \n",
       "Q 997 2584 770 2862 \n",
       "Q 544 3141 544 3541 \n",
       "Q 544 4100 942 4425 \n",
       "Q 1341 4750 2034 4750 \n",
       "Q 2731 4750 3128 4425 \n",
       "Q 3525 4100 3525 3541 \n",
       "Q 3525 3141 3298 2862 \n",
       "Q 3072 2584 2669 2484 \n",
       "Q 3125 2378 3379 2068 \n",
       "Q 3634 1759 3634 1313 \n",
       "Q 3634 634 3220 271 \n",
       "Q 2806 -91 2034 -91 \n",
       "Q 1263 -91 848 271 \n",
       "Q 434 634 434 1313 \n",
       "Q 434 1759 690 2068 \n",
       "Q 947 2378 1403 2484 \n",
       "z\n",
       "M 1172 3481 \n",
       "Q 1172 3119 1398 2916 \n",
       "Q 1625 2713 2034 2713 \n",
       "Q 2441 2713 2670 2916 \n",
       "Q 2900 3119 2900 3481 \n",
       "Q 2900 3844 2670 4047 \n",
       "Q 2441 4250 2034 4250 \n",
       "Q 1625 4250 1398 4047 \n",
       "Q 1172 3844 1172 3481 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-38\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 386.960938 173.52 \n",
       "L 386.960938 7.2 \n",
       "\" clip-path=\"url(#p6908287b77)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m34046a9be5\" x=\"386.960938\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 1000 -->\n",
       "      <g transform=\"translate(374.235937 188.118438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"190.869141\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- time -->\n",
       "     <g transform=\"translate(208.264844 201.796563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n",
       "Q 3544 3216 3844 3400 \n",
       "Q 4144 3584 4550 3584 \n",
       "Q 5097 3584 5394 3201 \n",
       "Q 5691 2819 5691 2113 \n",
       "L 5691 0 \n",
       "L 5113 0 \n",
       "L 5113 2094 \n",
       "Q 5113 2597 4934 2840 \n",
       "Q 4756 3084 4391 3084 \n",
       "Q 3944 3084 3684 2787 \n",
       "Q 3425 2491 3425 1978 \n",
       "L 3425 0 \n",
       "L 2847 0 \n",
       "L 2847 2094 \n",
       "Q 2847 2600 2669 2842 \n",
       "Q 2491 3084 2119 3084 \n",
       "Q 1678 3084 1418 2786 \n",
       "Q 1159 2488 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1356 3278 1631 3431 \n",
       "Q 1906 3584 2284 3584 \n",
       "Q 2666 3584 2933 3390 \n",
       "Q 3200 3197 3328 2828 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"66.992188\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"164.404297\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 52.160938 164.433887 \n",
       "L 386.960938 164.433887 \n",
       "\" clip-path=\"url(#p6908287b77)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"m609f5aa6c0\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m609f5aa6c0\" x=\"52.160938\" y=\"164.433887\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- −1.5 -->\n",
       "      <g transform=\"translate(20.878125 168.233106)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n",
       "L 4684 2272 \n",
       "L 4684 1741 \n",
       "L 678 1741 \n",
       "L 678 2272 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n",
       "L 1344 794 \n",
       "L 1344 0 \n",
       "L 684 0 \n",
       "L 684 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"83.789062\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"147.412109\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"179.199219\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 52.160938 140.309469 \n",
       "L 386.960938 140.309469 \n",
       "\" clip-path=\"url(#p6908287b77)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m609f5aa6c0\" x=\"52.160938\" y=\"140.309469\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- −1.0 -->\n",
       "      <g transform=\"translate(20.878125 144.108687)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"83.789062\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"147.412109\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"179.199219\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 52.160938 116.18505 \n",
       "L 386.960938 116.18505 \n",
       "\" clip-path=\"url(#p6908287b77)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m609f5aa6c0\" x=\"52.160938\" y=\"116.18505\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- −0.5 -->\n",
       "      <g transform=\"translate(20.878125 119.984269)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"83.789062\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"147.412109\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"179.199219\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <path d=\"M 52.160938 92.060631 \n",
       "L 386.960938 92.060631 \n",
       "\" clip-path=\"url(#p6908287b77)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m609f5aa6c0\" x=\"52.160938\" y=\"92.060631\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 0.0 -->\n",
       "      <g transform=\"translate(29.257812 95.85985)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <path d=\"M 52.160938 67.936212 \n",
       "L 386.960938 67.936212 \n",
       "\" clip-path=\"url(#p6908287b77)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m609f5aa6c0\" x=\"52.160938\" y=\"67.936212\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0.5 -->\n",
       "      <g transform=\"translate(29.257812 71.735431)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_21\">\n",
       "      <path d=\"M 52.160938 43.811793 \n",
       "L 386.960938 43.811793 \n",
       "\" clip-path=\"url(#p6908287b77)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_22\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m609f5aa6c0\" x=\"52.160938\" y=\"43.811793\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 1.0 -->\n",
       "      <g transform=\"translate(29.257812 47.611012)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_7\">\n",
       "     <g id=\"line2d_23\">\n",
       "      <path d=\"M 52.160938 19.687375 \n",
       "L 386.960938 19.687375 \n",
       "\" clip-path=\"url(#p6908287b77)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_24\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m609f5aa6c0\" x=\"52.160938\" y=\"19.687375\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 1.5 -->\n",
       "      <g transform=\"translate(29.257812 23.486593)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_14\">\n",
       "     <!-- x -->\n",
       "     <g transform=\"translate(14.798438 93.319375)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_25\">\n",
       "    <path d=\"M 52.160938 105.611947 \n",
       "L 52.496073 87.026764 \n",
       "L 52.831208 97.925394 \n",
       "L 53.166343 88.373368 \n",
       "L 53.501478 92.122679 \n",
       "L 53.836613 99.184428 \n",
       "L 54.171748 91.438558 \n",
       "L 54.842019 81.821461 \n",
       "L 55.177154 73.553499 \n",
       "L 55.512289 94.709178 \n",
       "L 55.847424 82.269602 \n",
       "L 56.182559 77.140946 \n",
       "L 56.517694 83.403274 \n",
       "L 56.852829 81.201675 \n",
       "L 57.187965 86.231106 \n",
       "L 57.5231 80.770048 \n",
       "L 57.858235 67.56994 \n",
       "L 58.528505 89.792422 \n",
       "L 58.86364 84.816995 \n",
       "L 59.198775 100.102794 \n",
       "L 59.53391 90.922272 \n",
       "L 59.869046 78.094588 \n",
       "L 60.204181 79.369213 \n",
       "L 60.539316 78.623734 \n",
       "L 61.209586 68.830376 \n",
       "L 61.544721 73.800086 \n",
       "L 61.879856 71.063621 \n",
       "L 62.214992 75.206554 \n",
       "L 62.550127 55.768033 \n",
       "L 62.885262 77.53077 \n",
       "L 63.220397 75.770226 \n",
       "L 63.555532 64.673713 \n",
       "L 63.890667 64.932166 \n",
       "L 64.560938 87.88931 \n",
       "L 64.896073 84.349478 \n",
       "L 65.231208 65.426753 \n",
       "L 65.566343 70.180386 \n",
       "L 65.901478 58.467745 \n",
       "L 66.236613 76.998732 \n",
       "L 66.571748 78.917609 \n",
       "L 66.906883 67.39097 \n",
       "L 67.242019 76.45564 \n",
       "L 67.577154 66.837362 \n",
       "L 67.912289 75.383409 \n",
       "L 68.247424 59.55577 \n",
       "L 68.582559 64.065314 \n",
       "L 68.917694 73.729396 \n",
       "L 69.252829 78.791445 \n",
       "L 69.587965 48.90323 \n",
       "L 69.9231 66.932956 \n",
       "L 70.258235 76.036169 \n",
       "L 70.59337 54.289988 \n",
       "L 70.928505 73.412326 \n",
       "L 71.26364 72.652985 \n",
       "L 71.598775 56.703175 \n",
       "L 71.93391 62.404411 \n",
       "L 72.269046 73.126086 \n",
       "L 72.604181 77.064162 \n",
       "L 72.939316 88.618029 \n",
       "L 73.609586 60.637546 \n",
       "L 73.944721 48.220605 \n",
       "L 74.279856 61.778492 \n",
       "L 74.614992 66.238262 \n",
       "L 74.950127 62.831533 \n",
       "L 75.285262 76.995044 \n",
       "L 75.620397 55.545195 \n",
       "L 75.955532 70.195015 \n",
       "L 76.290667 56.423397 \n",
       "L 76.625802 68.66343 \n",
       "L 76.960938 69.545336 \n",
       "L 77.296073 63.140457 \n",
       "L 77.631208 52.065701 \n",
       "L 77.966343 50.987307 \n",
       "L 78.301478 57.232654 \n",
       "L 78.636613 67.350941 \n",
       "L 79.306883 33.512937 \n",
       "L 79.977154 68.241701 \n",
       "L 80.312289 71.331773 \n",
       "L 80.647424 64.132402 \n",
       "L 80.982559 35.512795 \n",
       "L 81.652829 74.846008 \n",
       "L 82.3231 44.11027 \n",
       "L 82.658235 63.098964 \n",
       "L 82.99337 46.42935 \n",
       "L 83.328505 46.110162 \n",
       "L 83.66364 54.356759 \n",
       "L 83.998775 40.932965 \n",
       "L 84.669046 70.171797 \n",
       "L 85.339316 47.65686 \n",
       "L 86.009586 33.341502 \n",
       "L 86.344721 52.627931 \n",
       "L 86.679856 49.418062 \n",
       "L 87.014992 48.126513 \n",
       "L 87.350127 39.612338 \n",
       "L 87.685262 61.074555 \n",
       "L 88.020397 44.263291 \n",
       "L 88.355532 63.7495 \n",
       "L 88.690667 49.293886 \n",
       "L 89.025802 51.114026 \n",
       "L 89.360938 37.000406 \n",
       "L 89.696073 56.920196 \n",
       "L 90.031208 52.782643 \n",
       "L 90.366343 54.873384 \n",
       "L 90.701478 54.804464 \n",
       "L 91.036613 62.930689 \n",
       "L 91.371748 44.551061 \n",
       "L 91.706883 55.538327 \n",
       "L 92.042019 34.926742 \n",
       "L 92.377154 74.530792 \n",
       "L 92.712289 36.055998 \n",
       "L 93.047424 38.15417 \n",
       "L 93.382559 41.37025 \n",
       "L 93.717694 49.776227 \n",
       "L 94.052829 42.102351 \n",
       "L 94.387965 47.423994 \n",
       "L 94.7231 43.645908 \n",
       "L 95.058235 42.658115 \n",
       "L 95.39337 51.132391 \n",
       "L 95.728505 40.281814 \n",
       "L 96.06364 62.413656 \n",
       "L 96.398775 41.251915 \n",
       "L 96.73391 40.889275 \n",
       "L 97.069046 43.826946 \n",
       "L 97.404181 49.048417 \n",
       "L 97.739316 32.217606 \n",
       "L 98.074451 38.720938 \n",
       "L 98.409586 36.509992 \n",
       "L 98.744721 42.876847 \n",
       "L 99.414992 26.442251 \n",
       "L 99.750127 44.349213 \n",
       "L 100.085262 24.654114 \n",
       "L 100.420397 48.634176 \n",
       "L 100.755532 50.517999 \n",
       "L 101.090667 54.955972 \n",
       "L 101.425802 30.028615 \n",
       "L 101.760938 57.489589 \n",
       "L 102.096073 35.767464 \n",
       "L 102.431208 56.984324 \n",
       "L 102.766343 49.736765 \n",
       "L 103.101478 53.992075 \n",
       "L 103.436613 33.461615 \n",
       "L 104.106883 55.132124 \n",
       "L 104.442019 58.505755 \n",
       "L 105.112289 48.722919 \n",
       "L 105.447424 55.072335 \n",
       "L 105.782559 34.625531 \n",
       "L 106.117694 49.381165 \n",
       "L 106.452829 45.366222 \n",
       "L 106.787965 30.456784 \n",
       "L 107.1231 36.828867 \n",
       "L 107.458235 61.157495 \n",
       "L 107.79337 39.94416 \n",
       "L 108.128505 48.781437 \n",
       "L 108.46364 35.822594 \n",
       "L 108.798775 39.755619 \n",
       "L 109.13391 41.170137 \n",
       "L 109.469046 40.406816 \n",
       "L 109.804181 48.416673 \n",
       "L 110.139316 52.006409 \n",
       "L 110.809586 36.844385 \n",
       "L 111.144721 44.334305 \n",
       "L 111.479856 43.645765 \n",
       "L 111.814992 57.943232 \n",
       "L 112.150127 55.154976 \n",
       "L 112.485262 24.749891 \n",
       "L 112.820397 47.009917 \n",
       "L 113.155532 42.861456 \n",
       "L 113.490667 47.735431 \n",
       "L 113.825802 41.691857 \n",
       "L 114.160938 50.573414 \n",
       "L 114.496073 48.015183 \n",
       "L 114.831208 46.743448 \n",
       "L 115.166343 31.383298 \n",
       "L 115.836613 43.538248 \n",
       "L 116.171748 35.277947 \n",
       "L 116.506883 47.35512 \n",
       "L 116.842019 45.102109 \n",
       "L 117.177154 48.832182 \n",
       "L 117.512289 32.664716 \n",
       "L 117.847424 41.918244 \n",
       "L 118.182559 46.759841 \n",
       "L 118.517694 45.032096 \n",
       "L 118.852829 49.753525 \n",
       "L 119.187965 42.777446 \n",
       "L 119.5231 43.420516 \n",
       "L 119.858235 42.297104 \n",
       "L 120.19337 52.400519 \n",
       "L 120.528505 45.670982 \n",
       "L 120.86364 34.556407 \n",
       "L 121.198775 40.139131 \n",
       "L 121.53391 38.671686 \n",
       "L 122.204181 60.117212 \n",
       "L 122.874451 49.498857 \n",
       "L 123.209586 33.882209 \n",
       "L 123.879856 74.247008 \n",
       "L 124.214992 52.485095 \n",
       "L 124.550127 57.28867 \n",
       "L 124.885262 66.00216 \n",
       "L 125.220397 57.321817 \n",
       "L 125.555532 67.045428 \n",
       "L 125.890667 49.298562 \n",
       "L 126.225802 78.480028 \n",
       "L 126.560938 38.940573 \n",
       "L 126.896073 63.771813 \n",
       "L 127.231208 52.943196 \n",
       "L 127.566343 72.605643 \n",
       "L 127.901478 48.661618 \n",
       "L 128.236613 55.217816 \n",
       "L 128.571748 73.415888 \n",
       "L 128.906883 53.669827 \n",
       "L 129.242019 72.625856 \n",
       "L 129.577154 45.787414 \n",
       "L 129.912289 73.305527 \n",
       "L 130.582559 37.758165 \n",
       "L 130.917694 35.063742 \n",
       "L 131.587965 65.756921 \n",
       "L 131.9231 78.816839 \n",
       "L 132.258235 56.373381 \n",
       "L 132.59337 56.773343 \n",
       "L 132.928505 56.302551 \n",
       "L 133.598775 39.745645 \n",
       "L 134.269046 82.120959 \n",
       "L 134.604181 53.81519 \n",
       "L 134.939316 72.822594 \n",
       "L 135.274451 66.635245 \n",
       "L 135.609586 69.047454 \n",
       "L 135.944721 70.543215 \n",
       "L 136.279856 59.550648 \n",
       "L 136.614992 75.94289 \n",
       "L 136.950127 57.155377 \n",
       "L 137.285262 78.122691 \n",
       "L 137.620397 77.241596 \n",
       "L 137.955532 78.313943 \n",
       "L 138.290667 73.21288 \n",
       "L 138.625802 62.845958 \n",
       "L 138.960938 60.65064 \n",
       "L 139.296073 77.978321 \n",
       "L 139.631208 80.209138 \n",
       "L 139.966343 67.490285 \n",
       "L 140.301478 70.160301 \n",
       "L 140.636613 66.066691 \n",
       "L 140.971748 63.441875 \n",
       "L 141.306883 77.403069 \n",
       "L 141.642019 84.462039 \n",
       "L 141.977154 71.320199 \n",
       "L 142.312289 71.839416 \n",
       "L 142.647424 61.765755 \n",
       "L 142.982559 57.379288 \n",
       "L 143.317694 78.019196 \n",
       "L 143.652829 67.764038 \n",
       "L 143.987965 70.299355 \n",
       "L 144.3231 76.430718 \n",
       "L 144.658235 70.089684 \n",
       "L 144.99337 76.581478 \n",
       "L 145.328505 79.1972 \n",
       "L 145.66364 85.896258 \n",
       "L 145.998775 77.999342 \n",
       "L 146.33391 80.920864 \n",
       "L 146.669046 62.269622 \n",
       "L 147.004181 77.582639 \n",
       "L 147.339316 67.539747 \n",
       "L 147.674451 77.04422 \n",
       "L 148.009586 77.348077 \n",
       "L 148.344721 79.037141 \n",
       "L 148.679856 72.196803 \n",
       "L 149.014992 97.029881 \n",
       "L 149.350127 66.644088 \n",
       "L 149.685262 87.585314 \n",
       "L 150.020397 86.244215 \n",
       "L 150.355532 66.832977 \n",
       "L 150.690667 97.192735 \n",
       "L 151.025802 85.000628 \n",
       "L 151.360938 84.480349 \n",
       "L 151.696073 78.152942 \n",
       "L 152.031208 77.321797 \n",
       "L 152.366343 86.197021 \n",
       "L 152.701478 86.371927 \n",
       "L 153.036613 93.101495 \n",
       "L 153.371748 82.455425 \n",
       "L 153.706883 89.363467 \n",
       "L 154.042019 103.511546 \n",
       "L 154.377154 97.679887 \n",
       "L 154.712289 88.982592 \n",
       "L 155.047424 102.189302 \n",
       "L 155.382559 84.026723 \n",
       "L 155.717694 90.636532 \n",
       "L 156.052829 92.171041 \n",
       "L 156.387965 88.077323 \n",
       "L 157.39337 99.953436 \n",
       "L 157.728505 89.256108 \n",
       "L 158.06364 86.544955 \n",
       "L 158.398775 111.732532 \n",
       "L 158.73391 100.530903 \n",
       "L 159.069046 95.332426 \n",
       "L 159.404181 97.640862 \n",
       "L 159.739316 113.533161 \n",
       "L 160.074451 92.183167 \n",
       "L 160.409586 100.74188 \n",
       "L 160.744721 98.310132 \n",
       "L 161.079856 107.555461 \n",
       "L 161.414992 106.507069 \n",
       "L 161.750127 102.930336 \n",
       "L 162.085262 89.697025 \n",
       "L 162.420397 96.597823 \n",
       "L 162.755532 100.070084 \n",
       "L 163.090667 97.980419 \n",
       "L 163.425802 102.621301 \n",
       "L 163.760938 94.566775 \n",
       "L 164.096073 99.691332 \n",
       "L 164.431208 99.867618 \n",
       "L 164.766343 98.184301 \n",
       "L 165.101478 91.552276 \n",
       "L 165.436613 106.603659 \n",
       "L 165.771748 89.997534 \n",
       "L 166.106883 111.685536 \n",
       "L 166.442019 118.629825 \n",
       "L 166.777154 97.269259 \n",
       "L 167.112289 118.962769 \n",
       "L 167.447424 116.401003 \n",
       "L 167.782559 97.318193 \n",
       "L 168.117694 94.922265 \n",
       "L 168.452829 98.606217 \n",
       "L 168.787965 126.927276 \n",
       "L 169.1231 100.149035 \n",
       "L 169.458235 101.428196 \n",
       "L 169.79337 116.121997 \n",
       "L 170.128505 105.063659 \n",
       "L 170.46364 111.647018 \n",
       "L 170.798775 111.488405 \n",
       "L 171.13391 101.685794 \n",
       "L 171.469046 105.715693 \n",
       "L 171.804181 132.723312 \n",
       "L 172.139316 114.444807 \n",
       "L 172.474451 108.700362 \n",
       "L 172.809586 108.444354 \n",
       "L 173.144721 123.013224 \n",
       "L 173.479856 108.258363 \n",
       "L 173.814992 121.182351 \n",
       "L 174.150127 110.501783 \n",
       "L 174.485262 135.499417 \n",
       "L 174.820397 106.751682 \n",
       "L 175.155532 99.998297 \n",
       "L 175.490667 100.377671 \n",
       "L 175.825802 123.123101 \n",
       "L 176.160938 119.319858 \n",
       "L 176.496073 121.623432 \n",
       "L 176.831208 129.038766 \n",
       "L 177.166343 106.850994 \n",
       "L 177.501478 93.725271 \n",
       "L 177.836613 139.480895 \n",
       "L 178.171748 131.117345 \n",
       "L 178.506883 127.566654 \n",
       "L 178.842019 127.126245 \n",
       "L 179.177154 123.379691 \n",
       "L 179.512289 134.806279 \n",
       "L 179.847424 119.062308 \n",
       "L 180.182559 113.096318 \n",
       "L 180.517694 118.223864 \n",
       "L 180.852829 131.464081 \n",
       "L 181.187965 131.01049 \n",
       "L 181.5231 115.042888 \n",
       "L 181.858235 127.050604 \n",
       "L 182.19337 130.755427 \n",
       "L 182.528505 115.909833 \n",
       "L 182.86364 117.586017 \n",
       "L 183.198775 125.623792 \n",
       "L 183.53391 129.831576 \n",
       "L 183.869046 128.438662 \n",
       "L 184.204181 121.213373 \n",
       "L 184.539316 130.279936 \n",
       "L 184.874451 136.228147 \n",
       "L 185.209586 122.0721 \n",
       "L 185.544721 149.087279 \n",
       "L 185.879856 119.789485 \n",
       "L 186.214992 114.628442 \n",
       "L 186.550127 130.070908 \n",
       "L 186.885262 118.359495 \n",
       "L 187.220397 141.011056 \n",
       "L 187.555532 130.827974 \n",
       "L 187.890667 129.706491 \n",
       "L 188.225802 117.177378 \n",
       "L 188.560938 138.165338 \n",
       "L 188.896073 128.119629 \n",
       "L 189.231208 153.49233 \n",
       "L 189.566343 132.051518 \n",
       "L 189.901478 131.64897 \n",
       "L 190.236613 135.825332 \n",
       "L 190.571748 137.433913 \n",
       "L 190.906883 148.635724 \n",
       "L 191.242019 148.982863 \n",
       "L 191.577154 134.932773 \n",
       "L 191.912289 142.706194 \n",
       "L 192.247424 129.187252 \n",
       "L 192.582559 141.773474 \n",
       "L 192.917694 135.815068 \n",
       "L 193.252829 143.55724 \n",
       "L 193.587965 136.501733 \n",
       "L 193.9231 135.492602 \n",
       "L 194.258235 135.93104 \n",
       "L 194.59337 136.872592 \n",
       "L 194.928505 157.965112 \n",
       "L 195.26364 128.736589 \n",
       "L 195.598775 116.038976 \n",
       "L 195.93391 143.707285 \n",
       "L 196.269046 120.222083 \n",
       "L 196.604181 136.409548 \n",
       "L 196.939316 145.299732 \n",
       "L 197.274451 139.467873 \n",
       "L 197.609586 127.712192 \n",
       "L 197.944721 130.909582 \n",
       "L 198.279856 147.916622 \n",
       "L 198.614992 140.955702 \n",
       "L 198.950127 159.742931 \n",
       "L 199.285262 146.466081 \n",
       "L 199.620397 141.204676 \n",
       "L 199.955532 117.610427 \n",
       "L 200.290667 152.437075 \n",
       "L 200.960938 136.213297 \n",
       "L 201.296073 148.903092 \n",
       "L 201.631208 140.339907 \n",
       "L 201.966343 143.651793 \n",
       "L 202.301478 139.929258 \n",
       "L 202.636613 138.573338 \n",
       "L 202.971748 136.555817 \n",
       "L 203.306883 154.067713 \n",
       "L 203.642019 135.466034 \n",
       "L 203.977154 129.589449 \n",
       "L 204.312289 137.024411 \n",
       "L 204.647424 126.506987 \n",
       "L 204.982559 138.281344 \n",
       "L 205.317694 140.661393 \n",
       "L 205.652829 139.518456 \n",
       "L 205.987965 122.76473 \n",
       "L 206.3231 158.816394 \n",
       "L 206.658235 127.530401 \n",
       "L 206.99337 154.257042 \n",
       "L 207.328505 128.021398 \n",
       "L 207.66364 128.825421 \n",
       "L 207.998775 134.539322 \n",
       "L 208.33391 143.300841 \n",
       "L 209.004181 149.109303 \n",
       "L 209.339316 141.905982 \n",
       "L 209.674451 141.817946 \n",
       "L 210.009586 139.784795 \n",
       "L 210.344721 153.251379 \n",
       "L 210.679856 148.898025 \n",
       "L 211.014992 141.821271 \n",
       "L 211.350127 129.377467 \n",
       "L 211.685262 146.907082 \n",
       "L 212.020397 139.546757 \n",
       "L 212.355532 137.928827 \n",
       "L 212.690667 151.177451 \n",
       "L 213.360938 134.276679 \n",
       "L 213.696073 165.96 \n",
       "L 214.701478 118.611696 \n",
       "L 215.036613 136.766252 \n",
       "L 215.371748 147.408971 \n",
       "L 215.706883 136.590764 \n",
       "L 216.042019 141.117653 \n",
       "L 216.377154 151.482849 \n",
       "L 216.712289 125.32929 \n",
       "L 217.047424 145.246909 \n",
       "L 217.382559 143.69707 \n",
       "L 217.717694 135.298907 \n",
       "L 218.052829 141.639321 \n",
       "L 218.387965 128.026325 \n",
       "L 218.7231 147.931951 \n",
       "L 219.058235 151.053242 \n",
       "L 219.39337 123.666506 \n",
       "L 219.728505 128.894548 \n",
       "L 220.06364 143.695678 \n",
       "L 220.398775 144.637739 \n",
       "L 220.73391 124.111481 \n",
       "L 221.069046 132.283392 \n",
       "L 221.404181 134.423117 \n",
       "L 221.739316 145.173753 \n",
       "L 222.074451 143.769093 \n",
       "L 222.409586 137.097855 \n",
       "L 222.744721 127.485192 \n",
       "L 223.079856 131.006918 \n",
       "L 223.414992 141.092173 \n",
       "L 223.750127 125.91371 \n",
       "L 224.420397 161.117981 \n",
       "L 224.755532 130.924039 \n",
       "L 225.090667 131.613183 \n",
       "L 225.425802 158.60392 \n",
       "L 225.760938 137.034646 \n",
       "L 226.096073 146.831683 \n",
       "L 226.431208 117.904207 \n",
       "L 226.766343 114.500689 \n",
       "L 227.101478 137.825311 \n",
       "L 227.436613 138.621233 \n",
       "L 227.771748 131.235008 \n",
       "L 228.106883 133.976284 \n",
       "L 228.442019 135.030204 \n",
       "L 228.777154 143.229416 \n",
       "L 229.112289 146.547364 \n",
       "L 229.447424 136.076895 \n",
       "L 229.782559 139.740702 \n",
       "L 230.117694 137.543526 \n",
       "L 230.452829 129.331313 \n",
       "L 230.787965 116.901891 \n",
       "L 231.1231 128.834184 \n",
       "L 231.458235 134.833654 \n",
       "L 231.79337 148.468786 \n",
       "L 232.128505 132.565211 \n",
       "L 232.46364 134.52351 \n",
       "L 232.798775 134.036516 \n",
       "L 233.13391 111.902788 \n",
       "L 233.469046 128.197245 \n",
       "L 233.804181 124.908622 \n",
       "L 234.139316 134.752178 \n",
       "L 234.474451 129.364437 \n",
       "L 234.809586 111.661969 \n",
       "L 235.144721 127.563755 \n",
       "L 235.479856 116.721023 \n",
       "L 235.814992 116.331517 \n",
       "L 236.150127 126.855645 \n",
       "L 236.820397 114.595841 \n",
       "L 237.155532 114.930245 \n",
       "L 237.490667 137.409034 \n",
       "L 237.825802 132.157519 \n",
       "L 238.160938 113.186114 \n",
       "L 238.496073 125.696244 \n",
       "L 238.831208 128.79643 \n",
       "L 239.166343 134.696657 \n",
       "L 239.501478 132.927299 \n",
       "L 239.836613 116.241839 \n",
       "L 240.171748 122.167515 \n",
       "L 240.506883 117.993425 \n",
       "L 240.842019 130.163366 \n",
       "L 241.177154 126.896102 \n",
       "L 241.512289 109.491368 \n",
       "L 241.847424 111.201576 \n",
       "L 242.182559 108.440856 \n",
       "L 242.517694 116.005755 \n",
       "L 242.852829 105.12679 \n",
       "L 243.187965 119.38531 \n",
       "L 243.5231 113.637317 \n",
       "L 243.858235 118.892572 \n",
       "L 244.19337 118.238292 \n",
       "L 244.528505 106.364138 \n",
       "L 244.86364 132.195532 \n",
       "L 245.198775 115.162258 \n",
       "L 245.53391 121.495833 \n",
       "L 245.869046 119.581472 \n",
       "L 246.204181 99.979222 \n",
       "L 246.539316 107.589069 \n",
       "L 246.874451 98.445414 \n",
       "L 247.209586 111.149832 \n",
       "L 247.544721 113.774734 \n",
       "L 247.879856 96.423333 \n",
       "L 248.214992 87.923433 \n",
       "L 248.550127 106.595546 \n",
       "L 248.885262 115.160117 \n",
       "L 249.220397 116.228331 \n",
       "L 249.555532 115.317626 \n",
       "L 249.890667 116.645796 \n",
       "L 250.225802 114.805261 \n",
       "L 250.560938 121.693143 \n",
       "L 250.896073 113.017711 \n",
       "L 251.231208 126.802303 \n",
       "L 251.566343 105.585249 \n",
       "L 251.901478 112.610727 \n",
       "L 252.236613 103.221109 \n",
       "L 252.571748 108.085899 \n",
       "L 253.242019 124.616061 \n",
       "L 253.912289 95.149813 \n",
       "L 254.247424 87.482868 \n",
       "L 254.582559 99.511657 \n",
       "L 254.917694 118.587171 \n",
       "L 255.252829 104.442461 \n",
       "L 256.258235 86.858548 \n",
       "L 256.59337 103.870967 \n",
       "L 256.928505 103.409439 \n",
       "L 257.26364 99.342126 \n",
       "L 257.598775 92.067683 \n",
       "L 257.93391 90.016841 \n",
       "L 258.269046 85.61029 \n",
       "L 258.604181 104.235614 \n",
       "L 258.939316 96.9585 \n",
       "L 259.274451 83.215078 \n",
       "L 259.609586 99.046054 \n",
       "L 259.944721 102.955335 \n",
       "L 260.279856 74.245344 \n",
       "L 260.614992 107.63392 \n",
       "L 261.285262 82.991154 \n",
       "L 261.620397 95.343752 \n",
       "L 261.955532 92.502135 \n",
       "L 262.290667 80.414622 \n",
       "L 262.625802 101.874733 \n",
       "L 262.960938 89.143564 \n",
       "L 263.296073 101.363401 \n",
       "L 263.631208 85.473548 \n",
       "L 263.966343 96.135602 \n",
       "L 264.636613 79.326717 \n",
       "L 264.971748 86.332731 \n",
       "L 265.306883 95.790266 \n",
       "L 265.642019 90.066101 \n",
       "L 265.977154 100.828745 \n",
       "L 266.312289 98.375793 \n",
       "L 266.647424 79.144621 \n",
       "L 266.982559 89.183016 \n",
       "L 267.317694 95.296183 \n",
       "L 267.652829 88.972106 \n",
       "L 267.987965 91.763566 \n",
       "L 268.3231 73.986221 \n",
       "L 268.658235 80.779686 \n",
       "L 268.99337 76.151624 \n",
       "L 269.328505 93.998649 \n",
       "L 269.66364 94.594956 \n",
       "L 269.998775 63.330649 \n",
       "L 270.33391 87.512479 \n",
       "L 270.669046 65.818789 \n",
       "L 271.004181 73.011394 \n",
       "L 271.339316 72.829486 \n",
       "L 271.674451 76.627439 \n",
       "L 272.009586 73.396496 \n",
       "L 272.344721 82.181004 \n",
       "L 272.679856 94.980527 \n",
       "L 273.014992 79.863347 \n",
       "L 273.350127 83.15642 \n",
       "L 273.685262 84.800201 \n",
       "L 274.020397 58.389363 \n",
       "L 274.690667 85.451787 \n",
       "L 275.025802 80.559573 \n",
       "L 275.360937 72.559851 \n",
       "L 276.031208 75.021371 \n",
       "L 276.366343 77.511506 \n",
       "L 276.701478 65.802624 \n",
       "L 277.036613 76.33479 \n",
       "L 277.371748 57.624633 \n",
       "L 277.706883 80.88477 \n",
       "L 278.042019 72.594234 \n",
       "L 278.377154 67.314404 \n",
       "L 278.712289 71.047003 \n",
       "L 279.047424 56.467237 \n",
       "L 279.382559 72.693752 \n",
       "L 279.717694 56.156599 \n",
       "L 280.052829 52.699456 \n",
       "L 280.387965 71.225487 \n",
       "L 280.7231 61.160669 \n",
       "L 281.058235 72.75993 \n",
       "L 281.39337 57.794706 \n",
       "L 281.728505 58.894766 \n",
       "L 282.06364 73.240085 \n",
       "L 282.398775 58.29327 \n",
       "L 282.73391 71.985226 \n",
       "L 283.069046 73.321017 \n",
       "L 283.404181 64.599436 \n",
       "L 283.739316 65.871688 \n",
       "L 284.074451 68.996733 \n",
       "L 284.409586 78.859665 \n",
       "L 285.079856 54.746863 \n",
       "L 285.414992 69.810325 \n",
       "L 285.750127 66.584109 \n",
       "L 286.085262 48.020826 \n",
       "L 286.420397 47.861101 \n",
       "L 287.090667 60.815397 \n",
       "L 287.425802 59.95183 \n",
       "L 287.760938 44.650617 \n",
       "L 288.096073 79.41035 \n",
       "L 288.431208 52.806699 \n",
       "L 288.766343 66.609908 \n",
       "L 289.101478 43.136474 \n",
       "L 289.436613 40.097638 \n",
       "L 289.771748 41.039981 \n",
       "L 290.106883 62.466448 \n",
       "L 290.442019 50.900804 \n",
       "L 290.777154 64.933636 \n",
       "L 291.112289 65.703801 \n",
       "L 291.447424 49.751325 \n",
       "L 291.782559 48.986109 \n",
       "L 292.117694 42.188161 \n",
       "L 292.452829 51.382593 \n",
       "L 292.787965 47.355335 \n",
       "L 293.1231 76.074037 \n",
       "L 293.458235 56.021224 \n",
       "L 293.79337 58.854372 \n",
       "L 294.128505 59.249696 \n",
       "L 294.46364 54.850498 \n",
       "L 294.798775 34.770347 \n",
       "L 295.469046 68.770794 \n",
       "L 295.804181 65.133059 \n",
       "L 296.474451 60.753149 \n",
       "L 296.809586 47.777459 \n",
       "L 297.144721 41.661206 \n",
       "L 297.479856 54.274401 \n",
       "L 298.150127 25.037822 \n",
       "L 298.485262 53.549639 \n",
       "L 298.820397 57.291998 \n",
       "L 299.155532 48.926044 \n",
       "L 299.490667 36.012165 \n",
       "L 299.825802 51.492166 \n",
       "L 300.160938 55.334559 \n",
       "L 300.831208 39.819267 \n",
       "L 301.166343 50.371978 \n",
       "L 301.501478 54.104903 \n",
       "L 301.836613 47.704942 \n",
       "L 302.171748 64.176754 \n",
       "L 302.506883 48.530562 \n",
       "L 302.842019 43.733334 \n",
       "L 303.512289 55.615705 \n",
       "L 303.847424 44.142324 \n",
       "L 304.182559 41.16381 \n",
       "L 304.517694 46.278089 \n",
       "L 304.852829 46.945854 \n",
       "L 305.187965 36.005636 \n",
       "L 305.5231 40.918983 \n",
       "L 305.858235 43.880719 \n",
       "L 306.19337 43.347182 \n",
       "L 306.528505 48.065884 \n",
       "L 306.86364 36.611907 \n",
       "L 307.198775 62.949497 \n",
       "L 307.869046 34.786475 \n",
       "L 308.204181 54.01398 \n",
       "L 308.874451 40.012576 \n",
       "L 309.209586 58.949931 \n",
       "L 309.544721 41.989681 \n",
       "L 309.879856 43.328023 \n",
       "L 310.214992 57.393754 \n",
       "L 310.550127 51.541927 \n",
       "L 310.885262 35.915018 \n",
       "L 311.555532 46.253239 \n",
       "L 311.890667 47.868273 \n",
       "L 312.225802 26.574495 \n",
       "L 312.560938 42.143798 \n",
       "L 312.896073 37.752454 \n",
       "L 313.231208 48.12601 \n",
       "L 313.566343 46.029972 \n",
       "L 313.901478 50.023352 \n",
       "L 314.236613 37.185088 \n",
       "L 314.571748 38.91289 \n",
       "L 314.906883 44.238084 \n",
       "L 315.242019 41.049351 \n",
       "L 315.577154 14.76 \n",
       "L 315.912289 41.819775 \n",
       "L 316.247424 35.202761 \n",
       "L 316.582559 38.721082 \n",
       "L 316.917694 25.025536 \n",
       "L 317.252829 47.253898 \n",
       "L 317.587965 47.663687 \n",
       "L 317.9231 39.671362 \n",
       "L 318.258235 50.800667 \n",
       "L 318.59337 53.323859 \n",
       "L 318.928505 31.723799 \n",
       "L 319.26364 38.183383 \n",
       "L 319.598775 48.878612 \n",
       "L 319.93391 47.568934 \n",
       "L 320.269046 52.120394 \n",
       "L 320.604181 48.833542 \n",
       "L 320.939316 40.60399 \n",
       "L 321.274451 53.294907 \n",
       "L 321.609586 53.299681 \n",
       "L 321.944721 40.84153 \n",
       "L 322.279856 36.72575 \n",
       "L 322.614992 51.691989 \n",
       "L 322.950127 32.080744 \n",
       "L 323.285262 52.277111 \n",
       "L 323.620397 31.973573 \n",
       "L 323.955532 46.695542 \n",
       "L 324.290667 46.260615 \n",
       "L 324.625802 44.326871 \n",
       "L 324.960938 38.769465 \n",
       "L 325.296073 55.626806 \n",
       "L 325.631208 33.527599 \n",
       "L 325.966343 40.342707 \n",
       "L 326.301478 57.642622 \n",
       "L 326.636613 53.641738 \n",
       "L 326.971748 52.220281 \n",
       "L 327.306883 40.832609 \n",
       "L 327.642019 40.646449 \n",
       "L 327.977154 41.271073 \n",
       "L 328.312289 41.1215 \n",
       "L 328.647424 59.985244 \n",
       "L 329.317694 31.113278 \n",
       "L 329.652829 64.738917 \n",
       "L 329.987965 51.27552 \n",
       "L 330.3231 56.647786 \n",
       "L 330.658235 39.141854 \n",
       "L 330.99337 43.870697 \n",
       "L 331.328505 58.834756 \n",
       "L 331.66364 62.293653 \n",
       "L 331.998775 35.676662 \n",
       "L 332.33391 49.972257 \n",
       "L 332.669046 55.972599 \n",
       "L 333.004181 50.021256 \n",
       "L 333.339316 62.471677 \n",
       "L 333.674451 57.180771 \n",
       "L 334.009586 58.307192 \n",
       "L 334.344721 47.210832 \n",
       "L 334.679856 49.431777 \n",
       "L 335.014992 62.702642 \n",
       "L 335.350127 53.096232 \n",
       "L 335.685262 54.676718 \n",
       "L 336.020397 51.392918 \n",
       "L 336.355532 53.602454 \n",
       "L 336.690667 72.464536 \n",
       "L 337.025802 51.974876 \n",
       "L 337.360937 55.034219 \n",
       "L 337.696073 46.724948 \n",
       "L 338.031208 62.672541 \n",
       "L 338.366343 50.039791 \n",
       "L 338.701478 51.247351 \n",
       "L 339.036613 44.365677 \n",
       "L 339.371748 57.413278 \n",
       "L 339.706883 65.805054 \n",
       "L 340.042019 44.220297 \n",
       "L 340.377154 80.771039 \n",
       "L 340.712289 42.469994 \n",
       "L 341.047424 60.440233 \n",
       "L 341.382559 65.149635 \n",
       "L 341.717694 55.041814 \n",
       "L 342.052829 49.264935 \n",
       "L 342.387965 46.504514 \n",
       "L 342.7231 67.548694 \n",
       "L 343.39337 52.394124 \n",
       "L 344.06364 67.831364 \n",
       "L 344.398775 52.437207 \n",
       "L 344.73391 68.837741 \n",
       "L 345.069046 61.602398 \n",
       "L 345.404181 87.00912 \n",
       "L 345.739316 56.750325 \n",
       "L 346.074451 57.478077 \n",
       "L 346.409586 59.797664 \n",
       "L 346.744721 69.054136 \n",
       "L 347.414992 77.494907 \n",
       "L 347.750127 73.381223 \n",
       "L 348.420397 51.367072 \n",
       "L 348.755532 71.383154 \n",
       "L 349.090667 64.232243 \n",
       "L 349.425802 77.850599 \n",
       "L 349.760938 65.061249 \n",
       "L 350.096073 71.715205 \n",
       "L 350.431208 66.592903 \n",
       "L 350.766343 73.148804 \n",
       "L 351.101478 62.259591 \n",
       "L 351.436613 63.762211 \n",
       "L 351.771748 62.447163 \n",
       "L 352.106883 59.897675 \n",
       "L 352.442019 65.159698 \n",
       "L 352.777154 67.000004 \n",
       "L 353.112289 65.128138 \n",
       "L 353.447424 75.827835 \n",
       "L 353.782559 80.117053 \n",
       "L 354.117694 79.626746 \n",
       "L 354.452829 78.327966 \n",
       "L 354.787965 89.819699 \n",
       "L 355.1231 71.807301 \n",
       "L 355.458235 77.175549 \n",
       "L 355.79337 68.513376 \n",
       "L 356.128505 72.854656 \n",
       "L 356.46364 92.981711 \n",
       "L 356.798775 82.284014 \n",
       "L 357.13391 84.303537 \n",
       "L 358.139316 80.006719 \n",
       "L 358.474451 85.54437 \n",
       "L 358.809586 76.54493 \n",
       "L 359.144721 80.529747 \n",
       "L 359.479856 86.691838 \n",
       "L 359.814992 80.761264 \n",
       "L 360.150127 90.669399 \n",
       "L 360.485262 80.546514 \n",
       "L 360.820397 81.096667 \n",
       "L 361.155532 83.5527 \n",
       "L 361.490667 76.268153 \n",
       "L 361.825802 77.695047 \n",
       "L 362.160938 93.776426 \n",
       "L 362.496073 72.114279 \n",
       "L 362.831208 90.985157 \n",
       "L 363.166343 87.322855 \n",
       "L 363.501478 92.838829 \n",
       "L 363.836613 94.552408 \n",
       "L 364.171748 77.329577 \n",
       "L 364.506883 85.82083 \n",
       "L 364.842019 87.340957 \n",
       "L 365.177154 91.412248 \n",
       "L 365.512289 98.155708 \n",
       "L 365.847424 86.751275 \n",
       "L 366.182559 102.191471 \n",
       "L 366.517694 95.581068 \n",
       "L 366.852829 93.43472 \n",
       "L 367.187965 96.04051 \n",
       "L 367.5231 105.510066 \n",
       "L 367.858235 84.414364 \n",
       "L 368.19337 70.997258 \n",
       "L 368.528505 102.968257 \n",
       "L 368.86364 111.417619 \n",
       "L 369.198775 83.222807 \n",
       "L 369.53391 101.250299 \n",
       "L 369.869046 90.257391 \n",
       "L 370.204181 116.897974 \n",
       "L 370.539316 82.740908 \n",
       "L 371.544721 107.037203 \n",
       "L 371.879856 77.765663 \n",
       "L 372.214992 92.853224 \n",
       "L 372.550127 87.845542 \n",
       "L 372.885262 93.54923 \n",
       "L 373.220397 92.07708 \n",
       "L 373.555532 110.562756 \n",
       "L 373.890667 115.485694 \n",
       "L 374.560938 101.6236 \n",
       "L 374.896073 115.072459 \n",
       "L 375.231208 113.908651 \n",
       "L 375.566343 105.862205 \n",
       "L 375.901478 116.194232 \n",
       "L 376.236613 117.406989 \n",
       "L 376.571748 99.939875 \n",
       "L 377.242019 125.191428 \n",
       "L 377.577154 121.990486 \n",
       "L 377.912289 101.450332 \n",
       "L 378.247424 110.054101 \n",
       "L 378.582559 105.126599 \n",
       "L 378.917694 98.435479 \n",
       "L 379.252829 121.018309 \n",
       "L 379.587965 104.148759 \n",
       "L 379.9231 134.254046 \n",
       "L 380.258235 131.35722 \n",
       "L 380.59337 106.038102 \n",
       "L 380.928505 120.859438 \n",
       "L 381.26364 98.473247 \n",
       "L 381.598775 128.473494 \n",
       "L 381.93391 123.112734 \n",
       "L 382.269046 124.728852 \n",
       "L 382.604181 119.11904 \n",
       "L 382.939316 120.761383 \n",
       "L 383.274451 121.313757 \n",
       "L 383.609586 104.682847 \n",
       "L 383.944721 114.585038 \n",
       "L 384.279856 115.607568 \n",
       "L 384.614992 120.110937 \n",
       "L 384.950127 121.86855 \n",
       "L 385.285262 119.834708 \n",
       "L 385.620397 108.668054 \n",
       "L 386.290667 118.239753 \n",
       "L 386.960938 108.962568 \n",
       "L 386.960938 108.962568 \n",
       "\" clip-path=\"url(#p6908287b77)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 52.160938 173.52 \n",
       "L 52.160938 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 386.960938 173.52 \n",
       "L 386.960938 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 52.160938 173.52 \n",
       "L 386.960938 173.52 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 52.160938 7.2 \n",
       "L 386.960938 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p6908287b77\">\n",
       "   <rect x=\"52.160938\" y=\"7.2\" width=\"334.8\" height=\"166.32\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 600x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "# 导入必要的库\n",
    "%matplotlib inline\n",
    "import torch\n",
    "from torch import nn\n",
    "from d2l import torch as d2l\n",
    "\n",
    "# 定义总共产生1000个点的时间序列长度\n",
    "T = 1000\n",
    "\n",
    "# 生成包含从1到1000的浮点数的时间序列\n",
    "time = torch.arange(1, T + 1, dtype=torch.float32)\n",
    "\n",
    "# 创建一个正弦波形，并向其添加均值为0、标准差为0.2的噪声\n",
    "x = torch.sin(0.01 * time) + torch.normal(0, 0.2, (T,))\n",
    "\n",
    "# 使用 d2l.plot 函数将时间序列可视化为图表\n",
    "d2l.plot(time, [x], 'time', 'x', xlim=[1, 1000], figsize=(6, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "tau = 4  # tau定义了用于预测的时间步数\n",
    "features = torch.zeros((T - tau, tau))\n",
    "\n",
    "# 从原始时间序列 x 中提取特征\n",
    "for i in range(tau):\n",
    "    features[:, i] = x[i: T - tau + i]\n",
    "\n",
    "# 提取标签（即下一个时间步的值）\n",
    "labels = x[tau:].reshape((-1, 1))\n",
    "\n",
    "# 定义批量大小和训练样本数量\n",
    "batch_size, n_train = 16, 600\n",
    "\n",
    "# 创建训练迭代器，仅使用前 n_train 个样本进行训练\n",
    "train_iter = d2l.load_array((features[:n_train], labels[:n_train]),\n",
    "                            batch_size, is_train=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "class LSTM(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, output_size):\n",
    "        super(LSTM, self).__init__()\n",
    "\n",
    "        # 定义LSTM层，输入维度为input_size，隐藏单元数量为hidden_size\n",
    "        self.lstm = nn.LSTM(input_size, hidden_size)\n",
    "\n",
    "        # 定义线性层，将LSTM输出映射到输出维度为output_size\n",
    "        self.linear = nn.Linear(hidden_size, output_size)\n",
    "\n",
    "        # 保存隐藏单元数量\n",
    "        self.hidden_size = hidden_size\n",
    "\n",
    "    def forward(self, input):\n",
    "        # 将输入视图变形为 (sequence_length, batch_size, input_size)\n",
    "        lstm_out, _ = self.lstm(input.view(len(input), 1, -1))\n",
    "\n",
    "        # 将LSTM输出变形为 (sequence_length, hidden_size)，并通过线性层得到最终输出\n",
    "        output = self.linear(lstm_out.view(len(input), -1))\n",
    "\n",
    "        return output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "input_size = 4\n",
    "hidden_size = 200\n",
    "output_size = 1\n",
    "learning_rate = 0.01\n",
    "epochs = 50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LSTM(\n",
      "  (lstm): LSTM(4, 200)\n",
      "  (linear): Linear(in_features=200, out_features=1, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# 定义模型，input_size为输入维度，hidden_size为隐藏单元数量，output_size为输出维度\n",
    "model = LSTM(input_size, hidden_size, output_size)\n",
    "\n",
    "# 定义损失函数为均方误差损失\n",
    "criterion = nn.MSELoss()\n",
    "\n",
    "# 打印模型结构\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def train(model, train_iter, loss, epochs, learning_rate):\n",
    "    # 使用Adam优化器\n",
    "    trainer = torch.optim.Adam(model.parameters(), lr=learning_rate)\n",
    "\n",
    "    for epoch in range(epochs):\n",
    "        for X, y in train_iter:\n",
    "            # 将梯度归零\n",
    "            trainer.zero_grad()\n",
    "\n",
    "            # 前向传播\n",
    "            output = model(X)\n",
    "\n",
    "            # 计算损失\n",
    "            loss = criterion(output, y)\n",
    "\n",
    "            # 反向传播和优化\n",
    "            loss.backward()\n",
    "            trainer.step()\n",
    "\n",
    "        # 打印当前训练轮数和损失\n",
    "        if (epoch+1) % 5 == 0:\n",
    "            print('Epoch [{}/{}], Loss: {:.4f}'.format(epoch + 1, epochs, loss.item()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch [5/50], Loss: 0.0947\n",
      "Epoch [10/50], Loss: 0.0438\n",
      "Epoch [15/50], Loss: 0.0522\n",
      "Epoch [20/50], Loss: 0.0907\n",
      "Epoch [25/50], Loss: 0.0359\n",
      "Epoch [30/50], Loss: 0.0244\n",
      "Epoch [35/50], Loss: 0.0604\n",
      "Epoch [40/50], Loss: 0.0264\n",
      "Epoch [45/50], Loss: 0.0473\n",
      "Epoch [50/50], Loss: 0.0402\n"
     ]
    }
   ],
   "source": [
    "train(model, train_iter, criterion, 50, 0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"406.885938pt\" height=\"211.07625pt\" viewBox=\"0 0 406.885938 211.07625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2024-02-14T19:40:39.592266</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 211.07625 \n",
       "L 406.885938 211.07625 \n",
       "L 406.885938 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 52.160938 173.52 \n",
       "L 386.960938 173.52 \n",
       "L 386.960938 7.2 \n",
       "L 52.160938 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 118.852829 173.52 \n",
       "L 118.852829 7.2 \n",
       "\" clip-path=\"url(#pe84e13ccd4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"m5ac63aac92\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5ac63aac92\" x=\"118.852829\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 200 -->\n",
       "      <g transform=\"translate(109.309079 188.118438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 185.879856 173.52 \n",
       "L 185.879856 7.2 \n",
       "\" clip-path=\"url(#pe84e13ccd4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5ac63aac92\" x=\"185.879856\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 400 -->\n",
       "      <g transform=\"translate(176.336106 188.118438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 252.906883 173.52 \n",
       "L 252.906883 7.2 \n",
       "\" clip-path=\"url(#pe84e13ccd4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5ac63aac92\" x=\"252.906883\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 600 -->\n",
       "      <g transform=\"translate(243.363133 188.118438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n",
       "Q 1688 2584 1439 2293 \n",
       "Q 1191 2003 1191 1497 \n",
       "Q 1191 994 1439 701 \n",
       "Q 1688 409 2113 409 \n",
       "Q 2538 409 2786 701 \n",
       "Q 3034 994 3034 1497 \n",
       "Q 3034 2003 2786 2293 \n",
       "Q 2538 2584 2113 2584 \n",
       "z\n",
       "M 3366 4563 \n",
       "L 3366 3988 \n",
       "Q 3128 4100 2886 4159 \n",
       "Q 2644 4219 2406 4219 \n",
       "Q 1781 4219 1451 3797 \n",
       "Q 1122 3375 1075 2522 \n",
       "Q 1259 2794 1537 2939 \n",
       "Q 1816 3084 2150 3084 \n",
       "Q 2853 3084 3261 2657 \n",
       "Q 3669 2231 3669 1497 \n",
       "Q 3669 778 3244 343 \n",
       "Q 2819 -91 2113 -91 \n",
       "Q 1303 -91 875 529 \n",
       "Q 447 1150 447 2328 \n",
       "Q 447 3434 972 4092 \n",
       "Q 1497 4750 2381 4750 \n",
       "Q 2619 4750 2861 4703 \n",
       "Q 3103 4656 3366 4563 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 319.93391 173.52 \n",
       "L 319.93391 7.2 \n",
       "\" clip-path=\"url(#pe84e13ccd4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5ac63aac92\" x=\"319.93391\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 800 -->\n",
       "      <g transform=\"translate(310.39016 188.118438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n",
       "Q 1584 2216 1326 1975 \n",
       "Q 1069 1734 1069 1313 \n",
       "Q 1069 891 1326 650 \n",
       "Q 1584 409 2034 409 \n",
       "Q 2484 409 2743 651 \n",
       "Q 3003 894 3003 1313 \n",
       "Q 3003 1734 2745 1975 \n",
       "Q 2488 2216 2034 2216 \n",
       "z\n",
       "M 1403 2484 \n",
       "Q 997 2584 770 2862 \n",
       "Q 544 3141 544 3541 \n",
       "Q 544 4100 942 4425 \n",
       "Q 1341 4750 2034 4750 \n",
       "Q 2731 4750 3128 4425 \n",
       "Q 3525 4100 3525 3541 \n",
       "Q 3525 3141 3298 2862 \n",
       "Q 3072 2584 2669 2484 \n",
       "Q 3125 2378 3379 2068 \n",
       "Q 3634 1759 3634 1313 \n",
       "Q 3634 634 3220 271 \n",
       "Q 2806 -91 2034 -91 \n",
       "Q 1263 -91 848 271 \n",
       "Q 434 634 434 1313 \n",
       "Q 434 1759 690 2068 \n",
       "Q 947 2378 1403 2484 \n",
       "z\n",
       "M 1172 3481 \n",
       "Q 1172 3119 1398 2916 \n",
       "Q 1625 2713 2034 2713 \n",
       "Q 2441 2713 2670 2916 \n",
       "Q 2900 3119 2900 3481 \n",
       "Q 2900 3844 2670 4047 \n",
       "Q 2441 4250 2034 4250 \n",
       "Q 1625 4250 1398 4047 \n",
       "Q 1172 3844 1172 3481 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-38\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 386.960938 173.52 \n",
       "L 386.960938 7.2 \n",
       "\" clip-path=\"url(#pe84e13ccd4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5ac63aac92\" x=\"386.960938\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 1000 -->\n",
       "      <g transform=\"translate(374.235937 188.118438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"190.869141\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- time -->\n",
       "     <g transform=\"translate(208.264844 201.796563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n",
       "Q 3544 3216 3844 3400 \n",
       "Q 4144 3584 4550 3584 \n",
       "Q 5097 3584 5394 3201 \n",
       "Q 5691 2819 5691 2113 \n",
       "L 5691 0 \n",
       "L 5113 0 \n",
       "L 5113 2094 \n",
       "Q 5113 2597 4934 2840 \n",
       "Q 4756 3084 4391 3084 \n",
       "Q 3944 3084 3684 2787 \n",
       "Q 3425 2491 3425 1978 \n",
       "L 3425 0 \n",
       "L 2847 0 \n",
       "L 2847 2094 \n",
       "Q 2847 2600 2669 2842 \n",
       "Q 2491 3084 2119 3084 \n",
       "Q 1678 3084 1418 2786 \n",
       "Q 1159 2488 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1356 3278 1631 3431 \n",
       "Q 1906 3584 2284 3584 \n",
       "Q 2666 3584 2933 3390 \n",
       "Q 3200 3197 3328 2828 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"66.992188\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"164.404297\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 52.160938 164.433887 \n",
       "L 386.960938 164.433887 \n",
       "\" clip-path=\"url(#pe84e13ccd4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"m6c8a384c68\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m6c8a384c68\" x=\"52.160938\" y=\"164.433887\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- −1.5 -->\n",
       "      <g transform=\"translate(20.878125 168.233106)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n",
       "L 4684 2272 \n",
       "L 4684 1741 \n",
       "L 678 1741 \n",
       "L 678 2272 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n",
       "L 1344 794 \n",
       "L 1344 0 \n",
       "L 684 0 \n",
       "L 684 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"83.789062\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"147.412109\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"179.199219\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 52.160938 140.309469 \n",
       "L 386.960938 140.309469 \n",
       "\" clip-path=\"url(#pe84e13ccd4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m6c8a384c68\" x=\"52.160938\" y=\"140.309469\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- −1.0 -->\n",
       "      <g transform=\"translate(20.878125 144.108687)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"83.789062\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"147.412109\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"179.199219\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 52.160938 116.18505 \n",
       "L 386.960938 116.18505 \n",
       "\" clip-path=\"url(#pe84e13ccd4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m6c8a384c68\" x=\"52.160938\" y=\"116.18505\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- −0.5 -->\n",
       "      <g transform=\"translate(20.878125 119.984269)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"83.789062\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"147.412109\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"179.199219\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <path d=\"M 52.160938 92.060631 \n",
       "L 386.960938 92.060631 \n",
       "\" clip-path=\"url(#pe84e13ccd4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m6c8a384c68\" x=\"52.160938\" y=\"92.060631\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 0.0 -->\n",
       "      <g transform=\"translate(29.257812 95.85985)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <path d=\"M 52.160938 67.936212 \n",
       "L 386.960938 67.936212 \n",
       "\" clip-path=\"url(#pe84e13ccd4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m6c8a384c68\" x=\"52.160938\" y=\"67.936212\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0.5 -->\n",
       "      <g transform=\"translate(29.257812 71.735431)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_21\">\n",
       "      <path d=\"M 52.160938 43.811793 \n",
       "L 386.960938 43.811793 \n",
       "\" clip-path=\"url(#pe84e13ccd4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_22\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m6c8a384c68\" x=\"52.160938\" y=\"43.811793\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 1.0 -->\n",
       "      <g transform=\"translate(29.257812 47.611012)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_7\">\n",
       "     <g id=\"line2d_23\">\n",
       "      <path d=\"M 52.160938 19.687375 \n",
       "L 386.960938 19.687375 \n",
       "\" clip-path=\"url(#pe84e13ccd4)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_24\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m6c8a384c68\" x=\"52.160938\" y=\"19.687375\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 1.5 -->\n",
       "      <g transform=\"translate(29.257812 23.486593)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_14\">\n",
       "     <!-- x -->\n",
       "     <g transform=\"translate(14.798438 93.319375)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_25\">\n",
       "    <path d=\"M 52.160938 105.611947 \n",
       "L 52.496073 87.026764 \n",
       "L 52.831208 97.925394 \n",
       "L 53.166343 88.373368 \n",
       "L 53.501478 92.122679 \n",
       "L 53.836613 99.184428 \n",
       "L 54.171748 91.438558 \n",
       "L 54.842019 81.821461 \n",
       "L 55.177154 73.553499 \n",
       "L 55.512289 94.709178 \n",
       "L 55.847424 82.269602 \n",
       "L 56.182559 77.140946 \n",
       "L 56.517694 83.403274 \n",
       "L 56.852829 81.201675 \n",
       "L 57.187965 86.231106 \n",
       "L 57.5231 80.770048 \n",
       "L 57.858235 67.56994 \n",
       "L 58.528505 89.792422 \n",
       "L 58.86364 84.816995 \n",
       "L 59.198775 100.102794 \n",
       "L 59.53391 90.922272 \n",
       "L 59.869046 78.094588 \n",
       "L 60.204181 79.369213 \n",
       "L 60.539316 78.623734 \n",
       "L 61.209586 68.830376 \n",
       "L 61.544721 73.800086 \n",
       "L 61.879856 71.063621 \n",
       "L 62.214992 75.206554 \n",
       "L 62.550127 55.768033 \n",
       "L 62.885262 77.53077 \n",
       "L 63.220397 75.770226 \n",
       "L 63.555532 64.673713 \n",
       "L 63.890667 64.932166 \n",
       "L 64.560938 87.88931 \n",
       "L 64.896073 84.349478 \n",
       "L 65.231208 65.426753 \n",
       "L 65.566343 70.180386 \n",
       "L 65.901478 58.467745 \n",
       "L 66.236613 76.998732 \n",
       "L 66.571748 78.917609 \n",
       "L 66.906883 67.39097 \n",
       "L 67.242019 76.45564 \n",
       "L 67.577154 66.837362 \n",
       "L 67.912289 75.383409 \n",
       "L 68.247424 59.55577 \n",
       "L 68.582559 64.065314 \n",
       "L 68.917694 73.729396 \n",
       "L 69.252829 78.791445 \n",
       "L 69.587965 48.90323 \n",
       "L 69.9231 66.932956 \n",
       "L 70.258235 76.036169 \n",
       "L 70.59337 54.289988 \n",
       "L 70.928505 73.412326 \n",
       "L 71.26364 72.652985 \n",
       "L 71.598775 56.703175 \n",
       "L 71.93391 62.404411 \n",
       "L 72.269046 73.126086 \n",
       "L 72.604181 77.064162 \n",
       "L 72.939316 88.618029 \n",
       "L 73.609586 60.637546 \n",
       "L 73.944721 48.220605 \n",
       "L 74.279856 61.778492 \n",
       "L 74.614992 66.238262 \n",
       "L 74.950127 62.831533 \n",
       "L 75.285262 76.995044 \n",
       "L 75.620397 55.545195 \n",
       "L 75.955532 70.195015 \n",
       "L 76.290667 56.423397 \n",
       "L 76.625802 68.66343 \n",
       "L 76.960938 69.545336 \n",
       "L 77.296073 63.140457 \n",
       "L 77.631208 52.065701 \n",
       "L 77.966343 50.987307 \n",
       "L 78.301478 57.232654 \n",
       "L 78.636613 67.350941 \n",
       "L 79.306883 33.512937 \n",
       "L 79.977154 68.241701 \n",
       "L 80.312289 71.331773 \n",
       "L 80.647424 64.132402 \n",
       "L 80.982559 35.512795 \n",
       "L 81.652829 74.846008 \n",
       "L 82.3231 44.11027 \n",
       "L 82.658235 63.098964 \n",
       "L 82.99337 46.42935 \n",
       "L 83.328505 46.110162 \n",
       "L 83.66364 54.356759 \n",
       "L 83.998775 40.932965 \n",
       "L 84.669046 70.171797 \n",
       "L 85.339316 47.65686 \n",
       "L 86.009586 33.341502 \n",
       "L 86.344721 52.627931 \n",
       "L 86.679856 49.418062 \n",
       "L 87.014992 48.126513 \n",
       "L 87.350127 39.612338 \n",
       "L 87.685262 61.074555 \n",
       "L 88.020397 44.263291 \n",
       "L 88.355532 63.7495 \n",
       "L 88.690667 49.293886 \n",
       "L 89.025802 51.114026 \n",
       "L 89.360938 37.000406 \n",
       "L 89.696073 56.920196 \n",
       "L 90.031208 52.782643 \n",
       "L 90.366343 54.873384 \n",
       "L 90.701478 54.804464 \n",
       "L 91.036613 62.930689 \n",
       "L 91.371748 44.551061 \n",
       "L 91.706883 55.538327 \n",
       "L 92.042019 34.926742 \n",
       "L 92.377154 74.530792 \n",
       "L 92.712289 36.055998 \n",
       "L 93.047424 38.15417 \n",
       "L 93.382559 41.37025 \n",
       "L 93.717694 49.776227 \n",
       "L 94.052829 42.102351 \n",
       "L 94.387965 47.423994 \n",
       "L 94.7231 43.645908 \n",
       "L 95.058235 42.658115 \n",
       "L 95.39337 51.132391 \n",
       "L 95.728505 40.281814 \n",
       "L 96.06364 62.413656 \n",
       "L 96.398775 41.251915 \n",
       "L 96.73391 40.889275 \n",
       "L 97.069046 43.826946 \n",
       "L 97.404181 49.048417 \n",
       "L 97.739316 32.217606 \n",
       "L 98.074451 38.720938 \n",
       "L 98.409586 36.509992 \n",
       "L 98.744721 42.876847 \n",
       "L 99.414992 26.442251 \n",
       "L 99.750127 44.349213 \n",
       "L 100.085262 24.654114 \n",
       "L 100.420397 48.634176 \n",
       "L 100.755532 50.517999 \n",
       "L 101.090667 54.955972 \n",
       "L 101.425802 30.028615 \n",
       "L 101.760938 57.489589 \n",
       "L 102.096073 35.767464 \n",
       "L 102.431208 56.984324 \n",
       "L 102.766343 49.736765 \n",
       "L 103.101478 53.992075 \n",
       "L 103.436613 33.461615 \n",
       "L 104.106883 55.132124 \n",
       "L 104.442019 58.505755 \n",
       "L 105.112289 48.722919 \n",
       "L 105.447424 55.072335 \n",
       "L 105.782559 34.625531 \n",
       "L 106.117694 49.381165 \n",
       "L 106.452829 45.366222 \n",
       "L 106.787965 30.456784 \n",
       "L 107.1231 36.828867 \n",
       "L 107.458235 61.157495 \n",
       "L 107.79337 39.94416 \n",
       "L 108.128505 48.781437 \n",
       "L 108.46364 35.822594 \n",
       "L 108.798775 39.755619 \n",
       "L 109.13391 41.170137 \n",
       "L 109.469046 40.406816 \n",
       "L 109.804181 48.416673 \n",
       "L 110.139316 52.006409 \n",
       "L 110.809586 36.844385 \n",
       "L 111.144721 44.334305 \n",
       "L 111.479856 43.645765 \n",
       "L 111.814992 57.943232 \n",
       "L 112.150127 55.154976 \n",
       "L 112.485262 24.749891 \n",
       "L 112.820397 47.009917 \n",
       "L 113.155532 42.861456 \n",
       "L 113.490667 47.735431 \n",
       "L 113.825802 41.691857 \n",
       "L 114.160938 50.573414 \n",
       "L 114.496073 48.015183 \n",
       "L 114.831208 46.743448 \n",
       "L 115.166343 31.383298 \n",
       "L 115.836613 43.538248 \n",
       "L 116.171748 35.277947 \n",
       "L 116.506883 47.35512 \n",
       "L 116.842019 45.102109 \n",
       "L 117.177154 48.832182 \n",
       "L 117.512289 32.664716 \n",
       "L 117.847424 41.918244 \n",
       "L 118.182559 46.759841 \n",
       "L 118.517694 45.032096 \n",
       "L 118.852829 49.753525 \n",
       "L 119.187965 42.777446 \n",
       "L 119.5231 43.420516 \n",
       "L 119.858235 42.297104 \n",
       "L 120.19337 52.400519 \n",
       "L 120.528505 45.670982 \n",
       "L 120.86364 34.556407 \n",
       "L 121.198775 40.139131 \n",
       "L 121.53391 38.671686 \n",
       "L 122.204181 60.117212 \n",
       "L 122.874451 49.498857 \n",
       "L 123.209586 33.882209 \n",
       "L 123.879856 74.247008 \n",
       "L 124.214992 52.485095 \n",
       "L 124.550127 57.28867 \n",
       "L 124.885262 66.00216 \n",
       "L 125.220397 57.321817 \n",
       "L 125.555532 67.045428 \n",
       "L 125.890667 49.298562 \n",
       "L 126.225802 78.480028 \n",
       "L 126.560938 38.940573 \n",
       "L 126.896073 63.771813 \n",
       "L 127.231208 52.943196 \n",
       "L 127.566343 72.605643 \n",
       "L 127.901478 48.661618 \n",
       "L 128.236613 55.217816 \n",
       "L 128.571748 73.415888 \n",
       "L 128.906883 53.669827 \n",
       "L 129.242019 72.625856 \n",
       "L 129.577154 45.787414 \n",
       "L 129.912289 73.305527 \n",
       "L 130.582559 37.758165 \n",
       "L 130.917694 35.063742 \n",
       "L 131.587965 65.756921 \n",
       "L 131.9231 78.816839 \n",
       "L 132.258235 56.373381 \n",
       "L 132.59337 56.773343 \n",
       "L 132.928505 56.302551 \n",
       "L 133.598775 39.745645 \n",
       "L 134.269046 82.120959 \n",
       "L 134.604181 53.81519 \n",
       "L 134.939316 72.822594 \n",
       "L 135.274451 66.635245 \n",
       "L 135.609586 69.047454 \n",
       "L 135.944721 70.543215 \n",
       "L 136.279856 59.550648 \n",
       "L 136.614992 75.94289 \n",
       "L 136.950127 57.155377 \n",
       "L 137.285262 78.122691 \n",
       "L 137.620397 77.241596 \n",
       "L 137.955532 78.313943 \n",
       "L 138.290667 73.21288 \n",
       "L 138.625802 62.845958 \n",
       "L 138.960938 60.65064 \n",
       "L 139.296073 77.978321 \n",
       "L 139.631208 80.209138 \n",
       "L 139.966343 67.490285 \n",
       "L 140.301478 70.160301 \n",
       "L 140.636613 66.066691 \n",
       "L 140.971748 63.441875 \n",
       "L 141.306883 77.403069 \n",
       "L 141.642019 84.462039 \n",
       "L 141.977154 71.320199 \n",
       "L 142.312289 71.839416 \n",
       "L 142.647424 61.765755 \n",
       "L 142.982559 57.379288 \n",
       "L 143.317694 78.019196 \n",
       "L 143.652829 67.764038 \n",
       "L 143.987965 70.299355 \n",
       "L 144.3231 76.430718 \n",
       "L 144.658235 70.089684 \n",
       "L 144.99337 76.581478 \n",
       "L 145.328505 79.1972 \n",
       "L 145.66364 85.896258 \n",
       "L 145.998775 77.999342 \n",
       "L 146.33391 80.920864 \n",
       "L 146.669046 62.269622 \n",
       "L 147.004181 77.582639 \n",
       "L 147.339316 67.539747 \n",
       "L 147.674451 77.04422 \n",
       "L 148.009586 77.348077 \n",
       "L 148.344721 79.037141 \n",
       "L 148.679856 72.196803 \n",
       "L 149.014992 97.029881 \n",
       "L 149.350127 66.644088 \n",
       "L 149.685262 87.585314 \n",
       "L 150.020397 86.244215 \n",
       "L 150.355532 66.832977 \n",
       "L 150.690667 97.192735 \n",
       "L 151.025802 85.000628 \n",
       "L 151.360938 84.480349 \n",
       "L 151.696073 78.152942 \n",
       "L 152.031208 77.321797 \n",
       "L 152.366343 86.197021 \n",
       "L 152.701478 86.371927 \n",
       "L 153.036613 93.101495 \n",
       "L 153.371748 82.455425 \n",
       "L 153.706883 89.363467 \n",
       "L 154.042019 103.511546 \n",
       "L 154.377154 97.679887 \n",
       "L 154.712289 88.982592 \n",
       "L 155.047424 102.189302 \n",
       "L 155.382559 84.026723 \n",
       "L 155.717694 90.636532 \n",
       "L 156.052829 92.171041 \n",
       "L 156.387965 88.077323 \n",
       "L 157.39337 99.953436 \n",
       "L 157.728505 89.256108 \n",
       "L 158.06364 86.544955 \n",
       "L 158.398775 111.732532 \n",
       "L 158.73391 100.530903 \n",
       "L 159.069046 95.332426 \n",
       "L 159.404181 97.640862 \n",
       "L 159.739316 113.533161 \n",
       "L 160.074451 92.183167 \n",
       "L 160.409586 100.74188 \n",
       "L 160.744721 98.310132 \n",
       "L 161.079856 107.555461 \n",
       "L 161.414992 106.507069 \n",
       "L 161.750127 102.930336 \n",
       "L 162.085262 89.697025 \n",
       "L 162.420397 96.597823 \n",
       "L 162.755532 100.070084 \n",
       "L 163.090667 97.980419 \n",
       "L 163.425802 102.621301 \n",
       "L 163.760938 94.566775 \n",
       "L 164.096073 99.691332 \n",
       "L 164.431208 99.867618 \n",
       "L 164.766343 98.184301 \n",
       "L 165.101478 91.552276 \n",
       "L 165.436613 106.603659 \n",
       "L 165.771748 89.997534 \n",
       "L 166.106883 111.685536 \n",
       "L 166.442019 118.629825 \n",
       "L 166.777154 97.269259 \n",
       "L 167.112289 118.962769 \n",
       "L 167.447424 116.401003 \n",
       "L 167.782559 97.318193 \n",
       "L 168.117694 94.922265 \n",
       "L 168.452829 98.606217 \n",
       "L 168.787965 126.927276 \n",
       "L 169.1231 100.149035 \n",
       "L 169.458235 101.428196 \n",
       "L 169.79337 116.121997 \n",
       "L 170.128505 105.063659 \n",
       "L 170.46364 111.647018 \n",
       "L 170.798775 111.488405 \n",
       "L 171.13391 101.685794 \n",
       "L 171.469046 105.715693 \n",
       "L 171.804181 132.723312 \n",
       "L 172.139316 114.444807 \n",
       "L 172.474451 108.700362 \n",
       "L 172.809586 108.444354 \n",
       "L 173.144721 123.013224 \n",
       "L 173.479856 108.258363 \n",
       "L 173.814992 121.182351 \n",
       "L 174.150127 110.501783 \n",
       "L 174.485262 135.499417 \n",
       "L 174.820397 106.751682 \n",
       "L 175.155532 99.998297 \n",
       "L 175.490667 100.377671 \n",
       "L 175.825802 123.123101 \n",
       "L 176.160938 119.319858 \n",
       "L 176.496073 121.623432 \n",
       "L 176.831208 129.038766 \n",
       "L 177.166343 106.850994 \n",
       "L 177.501478 93.725271 \n",
       "L 177.836613 139.480895 \n",
       "L 178.171748 131.117345 \n",
       "L 178.506883 127.566654 \n",
       "L 178.842019 127.126245 \n",
       "L 179.177154 123.379691 \n",
       "L 179.512289 134.806279 \n",
       "L 179.847424 119.062308 \n",
       "L 180.182559 113.096318 \n",
       "L 180.517694 118.223864 \n",
       "L 180.852829 131.464081 \n",
       "L 181.187965 131.01049 \n",
       "L 181.5231 115.042888 \n",
       "L 181.858235 127.050604 \n",
       "L 182.19337 130.755427 \n",
       "L 182.528505 115.909833 \n",
       "L 182.86364 117.586017 \n",
       "L 183.198775 125.623792 \n",
       "L 183.53391 129.831576 \n",
       "L 183.869046 128.438662 \n",
       "L 184.204181 121.213373 \n",
       "L 184.539316 130.279936 \n",
       "L 184.874451 136.228147 \n",
       "L 185.209586 122.0721 \n",
       "L 185.544721 149.087279 \n",
       "L 185.879856 119.789485 \n",
       "L 186.214992 114.628442 \n",
       "L 186.550127 130.070908 \n",
       "L 186.885262 118.359495 \n",
       "L 187.220397 141.011056 \n",
       "L 187.555532 130.827974 \n",
       "L 187.890667 129.706491 \n",
       "L 188.225802 117.177378 \n",
       "L 188.560938 138.165338 \n",
       "L 188.896073 128.119629 \n",
       "L 189.231208 153.49233 \n",
       "L 189.566343 132.051518 \n",
       "L 189.901478 131.64897 \n",
       "L 190.236613 135.825332 \n",
       "L 190.571748 137.433913 \n",
       "L 190.906883 148.635724 \n",
       "L 191.242019 148.982863 \n",
       "L 191.577154 134.932773 \n",
       "L 191.912289 142.706194 \n",
       "L 192.247424 129.187252 \n",
       "L 192.582559 141.773474 \n",
       "L 192.917694 135.815068 \n",
       "L 193.252829 143.55724 \n",
       "L 193.587965 136.501733 \n",
       "L 193.9231 135.492602 \n",
       "L 194.258235 135.93104 \n",
       "L 194.59337 136.872592 \n",
       "L 194.928505 157.965112 \n",
       "L 195.26364 128.736589 \n",
       "L 195.598775 116.038976 \n",
       "L 195.93391 143.707285 \n",
       "L 196.269046 120.222083 \n",
       "L 196.604181 136.409548 \n",
       "L 196.939316 145.299732 \n",
       "L 197.274451 139.467873 \n",
       "L 197.609586 127.712192 \n",
       "L 197.944721 130.909582 \n",
       "L 198.279856 147.916622 \n",
       "L 198.614992 140.955702 \n",
       "L 198.950127 159.742931 \n",
       "L 199.285262 146.466081 \n",
       "L 199.620397 141.204676 \n",
       "L 199.955532 117.610427 \n",
       "L 200.290667 152.437075 \n",
       "L 200.960938 136.213297 \n",
       "L 201.296073 148.903092 \n",
       "L 201.631208 140.339907 \n",
       "L 201.966343 143.651793 \n",
       "L 202.301478 139.929258 \n",
       "L 202.636613 138.573338 \n",
       "L 202.971748 136.555817 \n",
       "L 203.306883 154.067713 \n",
       "L 203.642019 135.466034 \n",
       "L 203.977154 129.589449 \n",
       "L 204.312289 137.024411 \n",
       "L 204.647424 126.506987 \n",
       "L 204.982559 138.281344 \n",
       "L 205.317694 140.661393 \n",
       "L 205.652829 139.518456 \n",
       "L 205.987965 122.76473 \n",
       "L 206.3231 158.816394 \n",
       "L 206.658235 127.530401 \n",
       "L 206.99337 154.257042 \n",
       "L 207.328505 128.021398 \n",
       "L 207.66364 128.825421 \n",
       "L 207.998775 134.539322 \n",
       "L 208.33391 143.300841 \n",
       "L 209.004181 149.109303 \n",
       "L 209.339316 141.905982 \n",
       "L 209.674451 141.817946 \n",
       "L 210.009586 139.784795 \n",
       "L 210.344721 153.251379 \n",
       "L 210.679856 148.898025 \n",
       "L 211.014992 141.821271 \n",
       "L 211.350127 129.377467 \n",
       "L 211.685262 146.907082 \n",
       "L 212.020397 139.546757 \n",
       "L 212.355532 137.928827 \n",
       "L 212.690667 151.177451 \n",
       "L 213.360938 134.276679 \n",
       "L 213.696073 165.96 \n",
       "L 214.701478 118.611696 \n",
       "L 215.036613 136.766252 \n",
       "L 215.371748 147.408971 \n",
       "L 215.706883 136.590764 \n",
       "L 216.042019 141.117653 \n",
       "L 216.377154 151.482849 \n",
       "L 216.712289 125.32929 \n",
       "L 217.047424 145.246909 \n",
       "L 217.382559 143.69707 \n",
       "L 217.717694 135.298907 \n",
       "L 218.052829 141.639321 \n",
       "L 218.387965 128.026325 \n",
       "L 218.7231 147.931951 \n",
       "L 219.058235 151.053242 \n",
       "L 219.39337 123.666506 \n",
       "L 219.728505 128.894548 \n",
       "L 220.06364 143.695678 \n",
       "L 220.398775 144.637739 \n",
       "L 220.73391 124.111481 \n",
       "L 221.069046 132.283392 \n",
       "L 221.404181 134.423117 \n",
       "L 221.739316 145.173753 \n",
       "L 222.074451 143.769093 \n",
       "L 222.409586 137.097855 \n",
       "L 222.744721 127.485192 \n",
       "L 223.079856 131.006918 \n",
       "L 223.414992 141.092173 \n",
       "L 223.750127 125.91371 \n",
       "L 224.420397 161.117981 \n",
       "L 224.755532 130.924039 \n",
       "L 225.090667 131.613183 \n",
       "L 225.425802 158.60392 \n",
       "L 225.760938 137.034646 \n",
       "L 226.096073 146.831683 \n",
       "L 226.431208 117.904207 \n",
       "L 226.766343 114.500689 \n",
       "L 227.101478 137.825311 \n",
       "L 227.436613 138.621233 \n",
       "L 227.771748 131.235008 \n",
       "L 228.106883 133.976284 \n",
       "L 228.442019 135.030204 \n",
       "L 228.777154 143.229416 \n",
       "L 229.112289 146.547364 \n",
       "L 229.447424 136.076895 \n",
       "L 229.782559 139.740702 \n",
       "L 230.117694 137.543526 \n",
       "L 230.452829 129.331313 \n",
       "L 230.787965 116.901891 \n",
       "L 231.1231 128.834184 \n",
       "L 231.458235 134.833654 \n",
       "L 231.79337 148.468786 \n",
       "L 232.128505 132.565211 \n",
       "L 232.46364 134.52351 \n",
       "L 232.798775 134.036516 \n",
       "L 233.13391 111.902788 \n",
       "L 233.469046 128.197245 \n",
       "L 233.804181 124.908622 \n",
       "L 234.139316 134.752178 \n",
       "L 234.474451 129.364437 \n",
       "L 234.809586 111.661969 \n",
       "L 235.144721 127.563755 \n",
       "L 235.479856 116.721023 \n",
       "L 235.814992 116.331517 \n",
       "L 236.150127 126.855645 \n",
       "L 236.820397 114.595841 \n",
       "L 237.155532 114.930245 \n",
       "L 237.490667 137.409034 \n",
       "L 237.825802 132.157519 \n",
       "L 238.160938 113.186114 \n",
       "L 238.496073 125.696244 \n",
       "L 238.831208 128.79643 \n",
       "L 239.166343 134.696657 \n",
       "L 239.501478 132.927299 \n",
       "L 239.836613 116.241839 \n",
       "L 240.171748 122.167515 \n",
       "L 240.506883 117.993425 \n",
       "L 240.842019 130.163366 \n",
       "L 241.177154 126.896102 \n",
       "L 241.512289 109.491368 \n",
       "L 241.847424 111.201576 \n",
       "L 242.182559 108.440856 \n",
       "L 242.517694 116.005755 \n",
       "L 242.852829 105.12679 \n",
       "L 243.187965 119.38531 \n",
       "L 243.5231 113.637317 \n",
       "L 243.858235 118.892572 \n",
       "L 244.19337 118.238292 \n",
       "L 244.528505 106.364138 \n",
       "L 244.86364 132.195532 \n",
       "L 245.198775 115.162258 \n",
       "L 245.53391 121.495833 \n",
       "L 245.869046 119.581472 \n",
       "L 246.204181 99.979222 \n",
       "L 246.539316 107.589069 \n",
       "L 246.874451 98.445414 \n",
       "L 247.209586 111.149832 \n",
       "L 247.544721 113.774734 \n",
       "L 247.879856 96.423333 \n",
       "L 248.214992 87.923433 \n",
       "L 248.550127 106.595546 \n",
       "L 248.885262 115.160117 \n",
       "L 249.220397 116.228331 \n",
       "L 249.555532 115.317626 \n",
       "L 249.890667 116.645796 \n",
       "L 250.225802 114.805261 \n",
       "L 250.560938 121.693143 \n",
       "L 250.896073 113.017711 \n",
       "L 251.231208 126.802303 \n",
       "L 251.566343 105.585249 \n",
       "L 251.901478 112.610727 \n",
       "L 252.236613 103.221109 \n",
       "L 252.571748 108.085899 \n",
       "L 253.242019 124.616061 \n",
       "L 253.912289 95.149813 \n",
       "L 254.247424 87.482868 \n",
       "L 254.582559 99.511657 \n",
       "L 254.917694 118.587171 \n",
       "L 255.252829 104.442461 \n",
       "L 256.258235 86.858548 \n",
       "L 256.59337 103.870967 \n",
       "L 256.928505 103.409439 \n",
       "L 257.26364 99.342126 \n",
       "L 257.598775 92.067683 \n",
       "L 257.93391 90.016841 \n",
       "L 258.269046 85.61029 \n",
       "L 258.604181 104.235614 \n",
       "L 258.939316 96.9585 \n",
       "L 259.274451 83.215078 \n",
       "L 259.609586 99.046054 \n",
       "L 259.944721 102.955335 \n",
       "L 260.279856 74.245344 \n",
       "L 260.614992 107.63392 \n",
       "L 261.285262 82.991154 \n",
       "L 261.620397 95.343752 \n",
       "L 261.955532 92.502135 \n",
       "L 262.290667 80.414622 \n",
       "L 262.625802 101.874733 \n",
       "L 262.960938 89.143564 \n",
       "L 263.296073 101.363401 \n",
       "L 263.631208 85.473548 \n",
       "L 263.966343 96.135602 \n",
       "L 264.636613 79.326717 \n",
       "L 264.971748 86.332731 \n",
       "L 265.306883 95.790266 \n",
       "L 265.642019 90.066101 \n",
       "L 265.977154 100.828745 \n",
       "L 266.312289 98.375793 \n",
       "L 266.647424 79.144621 \n",
       "L 266.982559 89.183016 \n",
       "L 267.317694 95.296183 \n",
       "L 267.652829 88.972106 \n",
       "L 267.987965 91.763566 \n",
       "L 268.3231 73.986221 \n",
       "L 268.658235 80.779686 \n",
       "L 268.99337 76.151624 \n",
       "L 269.328505 93.998649 \n",
       "L 269.66364 94.594956 \n",
       "L 269.998775 63.330649 \n",
       "L 270.33391 87.512479 \n",
       "L 270.669046 65.818789 \n",
       "L 271.004181 73.011394 \n",
       "L 271.339316 72.829486 \n",
       "L 271.674451 76.627439 \n",
       "L 272.009586 73.396496 \n",
       "L 272.344721 82.181004 \n",
       "L 272.679856 94.980527 \n",
       "L 273.014992 79.863347 \n",
       "L 273.350127 83.15642 \n",
       "L 273.685262 84.800201 \n",
       "L 274.020397 58.389363 \n",
       "L 274.690667 85.451787 \n",
       "L 275.025802 80.559573 \n",
       "L 275.360937 72.559851 \n",
       "L 276.031208 75.021371 \n",
       "L 276.366343 77.511506 \n",
       "L 276.701478 65.802624 \n",
       "L 277.036613 76.33479 \n",
       "L 277.371748 57.624633 \n",
       "L 277.706883 80.88477 \n",
       "L 278.042019 72.594234 \n",
       "L 278.377154 67.314404 \n",
       "L 278.712289 71.047003 \n",
       "L 279.047424 56.467237 \n",
       "L 279.382559 72.693752 \n",
       "L 279.717694 56.156599 \n",
       "L 280.052829 52.699456 \n",
       "L 280.387965 71.225487 \n",
       "L 280.7231 61.160669 \n",
       "L 281.058235 72.75993 \n",
       "L 281.39337 57.794706 \n",
       "L 281.728505 58.894766 \n",
       "L 282.06364 73.240085 \n",
       "L 282.398775 58.29327 \n",
       "L 282.73391 71.985226 \n",
       "L 283.069046 73.321017 \n",
       "L 283.404181 64.599436 \n",
       "L 283.739316 65.871688 \n",
       "L 284.074451 68.996733 \n",
       "L 284.409586 78.859665 \n",
       "L 285.079856 54.746863 \n",
       "L 285.414992 69.810325 \n",
       "L 285.750127 66.584109 \n",
       "L 286.085262 48.020826 \n",
       "L 286.420397 47.861101 \n",
       "L 287.090667 60.815397 \n",
       "L 287.425802 59.95183 \n",
       "L 287.760938 44.650617 \n",
       "L 288.096073 79.41035 \n",
       "L 288.431208 52.806699 \n",
       "L 288.766343 66.609908 \n",
       "L 289.101478 43.136474 \n",
       "L 289.436613 40.097638 \n",
       "L 289.771748 41.039981 \n",
       "L 290.106883 62.466448 \n",
       "L 290.442019 50.900804 \n",
       "L 290.777154 64.933636 \n",
       "L 291.112289 65.703801 \n",
       "L 291.447424 49.751325 \n",
       "L 291.782559 48.986109 \n",
       "L 292.117694 42.188161 \n",
       "L 292.452829 51.382593 \n",
       "L 292.787965 47.355335 \n",
       "L 293.1231 76.074037 \n",
       "L 293.458235 56.021224 \n",
       "L 293.79337 58.854372 \n",
       "L 294.128505 59.249696 \n",
       "L 294.46364 54.850498 \n",
       "L 294.798775 34.770347 \n",
       "L 295.469046 68.770794 \n",
       "L 295.804181 65.133059 \n",
       "L 296.474451 60.753149 \n",
       "L 296.809586 47.777459 \n",
       "L 297.144721 41.661206 \n",
       "L 297.479856 54.274401 \n",
       "L 298.150127 25.037822 \n",
       "L 298.485262 53.549639 \n",
       "L 298.820397 57.291998 \n",
       "L 299.155532 48.926044 \n",
       "L 299.490667 36.012165 \n",
       "L 299.825802 51.492166 \n",
       "L 300.160938 55.334559 \n",
       "L 300.831208 39.819267 \n",
       "L 301.166343 50.371978 \n",
       "L 301.501478 54.104903 \n",
       "L 301.836613 47.704942 \n",
       "L 302.171748 64.176754 \n",
       "L 302.506883 48.530562 \n",
       "L 302.842019 43.733334 \n",
       "L 303.512289 55.615705 \n",
       "L 303.847424 44.142324 \n",
       "L 304.182559 41.16381 \n",
       "L 304.517694 46.278089 \n",
       "L 304.852829 46.945854 \n",
       "L 305.187965 36.005636 \n",
       "L 305.5231 40.918983 \n",
       "L 305.858235 43.880719 \n",
       "L 306.19337 43.347182 \n",
       "L 306.528505 48.065884 \n",
       "L 306.86364 36.611907 \n",
       "L 307.198775 62.949497 \n",
       "L 307.869046 34.786475 \n",
       "L 308.204181 54.01398 \n",
       "L 308.874451 40.012576 \n",
       "L 309.209586 58.949931 \n",
       "L 309.544721 41.989681 \n",
       "L 309.879856 43.328023 \n",
       "L 310.214992 57.393754 \n",
       "L 310.550127 51.541927 \n",
       "L 310.885262 35.915018 \n",
       "L 311.555532 46.253239 \n",
       "L 311.890667 47.868273 \n",
       "L 312.225802 26.574495 \n",
       "L 312.560938 42.143798 \n",
       "L 312.896073 37.752454 \n",
       "L 313.231208 48.12601 \n",
       "L 313.566343 46.029972 \n",
       "L 313.901478 50.023352 \n",
       "L 314.236613 37.185088 \n",
       "L 314.571748 38.91289 \n",
       "L 314.906883 44.238084 \n",
       "L 315.242019 41.049351 \n",
       "L 315.577154 14.76 \n",
       "L 315.912289 41.819775 \n",
       "L 316.247424 35.202761 \n",
       "L 316.582559 38.721082 \n",
       "L 316.917694 25.025536 \n",
       "L 317.252829 47.253898 \n",
       "L 317.587965 47.663687 \n",
       "L 317.9231 39.671362 \n",
       "L 318.258235 50.800667 \n",
       "L 318.59337 53.323859 \n",
       "L 318.928505 31.723799 \n",
       "L 319.26364 38.183383 \n",
       "L 319.598775 48.878612 \n",
       "L 319.93391 47.568934 \n",
       "L 320.269046 52.120394 \n",
       "L 320.604181 48.833542 \n",
       "L 320.939316 40.60399 \n",
       "L 321.274451 53.294907 \n",
       "L 321.609586 53.299681 \n",
       "L 321.944721 40.84153 \n",
       "L 322.279856 36.72575 \n",
       "L 322.614992 51.691989 \n",
       "L 322.950127 32.080744 \n",
       "L 323.285262 52.277111 \n",
       "L 323.620397 31.973573 \n",
       "L 323.955532 46.695542 \n",
       "L 324.290667 46.260615 \n",
       "L 324.625802 44.326871 \n",
       "L 324.960938 38.769465 \n",
       "L 325.296073 55.626806 \n",
       "L 325.631208 33.527599 \n",
       "L 325.966343 40.342707 \n",
       "L 326.301478 57.642622 \n",
       "L 326.636613 53.641738 \n",
       "L 326.971748 52.220281 \n",
       "L 327.306883 40.832609 \n",
       "L 327.642019 40.646449 \n",
       "L 327.977154 41.271073 \n",
       "L 328.312289 41.1215 \n",
       "L 328.647424 59.985244 \n",
       "L 329.317694 31.113278 \n",
       "L 329.652829 64.738917 \n",
       "L 329.987965 51.27552 \n",
       "L 330.3231 56.647786 \n",
       "L 330.658235 39.141854 \n",
       "L 330.99337 43.870697 \n",
       "L 331.328505 58.834756 \n",
       "L 331.66364 62.293653 \n",
       "L 331.998775 35.676662 \n",
       "L 332.33391 49.972257 \n",
       "L 332.669046 55.972599 \n",
       "L 333.004181 50.021256 \n",
       "L 333.339316 62.471677 \n",
       "L 333.674451 57.180771 \n",
       "L 334.009586 58.307192 \n",
       "L 334.344721 47.210832 \n",
       "L 334.679856 49.431777 \n",
       "L 335.014992 62.702642 \n",
       "L 335.350127 53.096232 \n",
       "L 335.685262 54.676718 \n",
       "L 336.020397 51.392918 \n",
       "L 336.355532 53.602454 \n",
       "L 336.690667 72.464536 \n",
       "L 337.025802 51.974876 \n",
       "L 337.360937 55.034219 \n",
       "L 337.696073 46.724948 \n",
       "L 338.031208 62.672541 \n",
       "L 338.366343 50.039791 \n",
       "L 338.701478 51.247351 \n",
       "L 339.036613 44.365677 \n",
       "L 339.371748 57.413278 \n",
       "L 339.706883 65.805054 \n",
       "L 340.042019 44.220297 \n",
       "L 340.377154 80.771039 \n",
       "L 340.712289 42.469994 \n",
       "L 341.047424 60.440233 \n",
       "L 341.382559 65.149635 \n",
       "L 341.717694 55.041814 \n",
       "L 342.052829 49.264935 \n",
       "L 342.387965 46.504514 \n",
       "L 342.7231 67.548694 \n",
       "L 343.39337 52.394124 \n",
       "L 344.06364 67.831364 \n",
       "L 344.398775 52.437207 \n",
       "L 344.73391 68.837741 \n",
       "L 345.069046 61.602398 \n",
       "L 345.404181 87.00912 \n",
       "L 345.739316 56.750325 \n",
       "L 346.074451 57.478077 \n",
       "L 346.409586 59.797664 \n",
       "L 346.744721 69.054136 \n",
       "L 347.414992 77.494907 \n",
       "L 347.750127 73.381223 \n",
       "L 348.420397 51.367072 \n",
       "L 348.755532 71.383154 \n",
       "L 349.090667 64.232243 \n",
       "L 349.425802 77.850599 \n",
       "L 349.760938 65.061249 \n",
       "L 350.096073 71.715205 \n",
       "L 350.431208 66.592903 \n",
       "L 350.766343 73.148804 \n",
       "L 351.101478 62.259591 \n",
       "L 351.436613 63.762211 \n",
       "L 351.771748 62.447163 \n",
       "L 352.106883 59.897675 \n",
       "L 352.442019 65.159698 \n",
       "L 352.777154 67.000004 \n",
       "L 353.112289 65.128138 \n",
       "L 353.447424 75.827835 \n",
       "L 353.782559 80.117053 \n",
       "L 354.117694 79.626746 \n",
       "L 354.452829 78.327966 \n",
       "L 354.787965 89.819699 \n",
       "L 355.1231 71.807301 \n",
       "L 355.458235 77.175549 \n",
       "L 355.79337 68.513376 \n",
       "L 356.128505 72.854656 \n",
       "L 356.46364 92.981711 \n",
       "L 356.798775 82.284014 \n",
       "L 357.13391 84.303537 \n",
       "L 358.139316 80.006719 \n",
       "L 358.474451 85.54437 \n",
       "L 358.809586 76.54493 \n",
       "L 359.144721 80.529747 \n",
       "L 359.479856 86.691838 \n",
       "L 359.814992 80.761264 \n",
       "L 360.150127 90.669399 \n",
       "L 360.485262 80.546514 \n",
       "L 360.820397 81.096667 \n",
       "L 361.155532 83.5527 \n",
       "L 361.490667 76.268153 \n",
       "L 361.825802 77.695047 \n",
       "L 362.160938 93.776426 \n",
       "L 362.496073 72.114279 \n",
       "L 362.831208 90.985157 \n",
       "L 363.166343 87.322855 \n",
       "L 363.501478 92.838829 \n",
       "L 363.836613 94.552408 \n",
       "L 364.171748 77.329577 \n",
       "L 364.506883 85.82083 \n",
       "L 364.842019 87.340957 \n",
       "L 365.177154 91.412248 \n",
       "L 365.512289 98.155708 \n",
       "L 365.847424 86.751275 \n",
       "L 366.182559 102.191471 \n",
       "L 366.517694 95.581068 \n",
       "L 366.852829 93.43472 \n",
       "L 367.187965 96.04051 \n",
       "L 367.5231 105.510066 \n",
       "L 367.858235 84.414364 \n",
       "L 368.19337 70.997258 \n",
       "L 368.528505 102.968257 \n",
       "L 368.86364 111.417619 \n",
       "L 369.198775 83.222807 \n",
       "L 369.53391 101.250299 \n",
       "L 369.869046 90.257391 \n",
       "L 370.204181 116.897974 \n",
       "L 370.539316 82.740908 \n",
       "L 371.544721 107.037203 \n",
       "L 371.879856 77.765663 \n",
       "L 372.214992 92.853224 \n",
       "L 372.550127 87.845542 \n",
       "L 372.885262 93.54923 \n",
       "L 373.220397 92.07708 \n",
       "L 373.555532 110.562756 \n",
       "L 373.890667 115.485694 \n",
       "L 374.560938 101.6236 \n",
       "L 374.896073 115.072459 \n",
       "L 375.231208 113.908651 \n",
       "L 375.566343 105.862205 \n",
       "L 375.901478 116.194232 \n",
       "L 376.236613 117.406989 \n",
       "L 376.571748 99.939875 \n",
       "L 377.242019 125.191428 \n",
       "L 377.577154 121.990486 \n",
       "L 377.912289 101.450332 \n",
       "L 378.247424 110.054101 \n",
       "L 378.582559 105.126599 \n",
       "L 378.917694 98.435479 \n",
       "L 379.252829 121.018309 \n",
       "L 379.587965 104.148759 \n",
       "L 379.9231 134.254046 \n",
       "L 380.258235 131.35722 \n",
       "L 380.59337 106.038102 \n",
       "L 380.928505 120.859438 \n",
       "L 381.26364 98.473247 \n",
       "L 381.598775 128.473494 \n",
       "L 381.93391 123.112734 \n",
       "L 382.269046 124.728852 \n",
       "L 382.604181 119.11904 \n",
       "L 382.939316 120.761383 \n",
       "L 383.274451 121.313757 \n",
       "L 383.609586 104.682847 \n",
       "L 383.944721 114.585038 \n",
       "L 384.279856 115.607568 \n",
       "L 384.614992 120.110937 \n",
       "L 384.950127 121.86855 \n",
       "L 385.285262 119.834708 \n",
       "L 385.620397 108.668054 \n",
       "L 386.290667 118.239753 \n",
       "L 386.960938 108.962568 \n",
       "L 386.960938 108.962568 \n",
       "\" clip-path=\"url(#pe84e13ccd4)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_26\">\n",
       "    <path d=\"M 53.501478 92.480482 \n",
       "L 53.836613 89.191988 \n",
       "L 54.171748 93.154049 \n",
       "L 54.506883 90.455928 \n",
       "L 54.842019 89.604574 \n",
       "L 55.177154 87.005298 \n",
       "L 55.512289 80.056558 \n",
       "L 55.847424 82.89398 \n",
       "L 56.182559 81.045267 \n",
       "L 56.517694 78.181366 \n",
       "L 56.852829 82.364741 \n",
       "L 57.187965 78.624545 \n",
       "L 57.5231 79.746027 \n",
       "L 57.858235 80.447453 \n",
       "L 58.19337 74.993586 \n",
       "L 58.528505 75.581149 \n",
       "L 58.86364 78.042173 \n",
       "L 59.198775 77.875261 \n",
       "L 59.53391 87.176862 \n",
       "L 59.869046 89.274227 \n",
       "L 60.204181 84.883346 \n",
       "L 60.539316 84.324584 \n",
       "L 61.209586 74.002172 \n",
       "L 61.544721 71.572135 \n",
       "L 61.879856 70.872658 \n",
       "L 62.214992 68.967893 \n",
       "L 62.550127 69.336966 \n",
       "L 62.885262 65.302182 \n",
       "L 63.220397 66.853329 \n",
       "L 63.555532 70.171739 \n",
       "L 63.890667 63.583606 \n",
       "L 64.225802 67.684888 \n",
       "L 64.560938 68.326425 \n",
       "L 65.231208 76.305482 \n",
       "L 65.566343 74.263685 \n",
       "L 65.901478 73.482196 \n",
       "L 66.236613 66.773447 \n",
       "L 66.571748 64.639232 \n",
       "L 66.906883 70.220314 \n",
       "L 67.242019 66.247838 \n",
       "L 67.577154 72.423919 \n",
       "L 67.912289 69.767942 \n",
       "L 68.582559 66.760566 \n",
       "L 68.917694 62.048348 \n",
       "L 69.252829 66.95539 \n",
       "L 69.587965 66.690096 \n",
       "L 69.9231 61.774104 \n",
       "L 70.258235 63.164102 \n",
       "L 70.59337 67.582706 \n",
       "L 70.928505 56.444593 \n",
       "L 71.26364 64.854495 \n",
       "L 71.598775 68.181393 \n",
       "L 71.93391 58.944518 \n",
       "L 72.269046 63.208652 \n",
       "L 72.604181 64.803535 \n",
       "L 72.939316 64.955524 \n",
       "L 73.274451 73.896157 \n",
       "L 73.609586 74.851296 \n",
       "L 74.614992 57.962814 \n",
       "L 74.950127 58.170048 \n",
       "L 75.285262 56.17589 \n",
       "L 75.620397 65.088081 \n",
       "L 75.955532 62.313732 \n",
       "L 76.290667 62.216128 \n",
       "L 76.625802 63.271256 \n",
       "L 76.960938 58.25848 \n",
       "L 77.296073 65.214172 \n",
       "L 77.631208 60.253596 \n",
       "L 77.966343 59.66071 \n",
       "L 78.301478 55.665949 \n",
       "L 78.636613 54.169101 \n",
       "L 78.971748 55.156744 \n",
       "L 79.306883 52.797485 \n",
       "L 79.642019 46.943717 \n",
       "L 79.977154 50.294123 \n",
       "L 80.312289 51.319011 \n",
       "L 80.647424 53.850911 \n",
       "L 80.982559 61.019405 \n",
       "L 81.317694 54.889071 \n",
       "L 81.652829 52.885038 \n",
       "L 81.987965 58.065447 \n",
       "L 82.3231 51.958567 \n",
       "L 82.658235 52.391714 \n",
       "L 82.99337 58.844723 \n",
       "L 83.66364 45.283193 \n",
       "L 83.998775 52.575475 \n",
       "L 84.33391 44.974487 \n",
       "L 84.669046 47.524971 \n",
       "L 85.004181 56.163788 \n",
       "L 85.339316 52.745291 \n",
       "L 85.674451 53.925574 \n",
       "L 86.009586 51.816922 \n",
       "L 86.344721 43.71773 \n",
       "L 86.679856 43.932668 \n",
       "L 87.014992 44.970789 \n",
       "L 87.350127 42.747578 \n",
       "L 87.685262 46.406953 \n",
       "L 88.020397 48.217741 \n",
       "L 88.355532 47.933287 \n",
       "L 88.690667 48.235586 \n",
       "L 89.025802 54.468938 \n",
       "L 89.360938 47.048246 \n",
       "L 89.696073 49.599184 \n",
       "L 90.031208 46.414221 \n",
       "L 90.366343 50.041418 \n",
       "L 90.701478 46.447816 \n",
       "L 91.036613 53.534846 \n",
       "L 91.371748 53.844285 \n",
       "L 91.706883 51.587561 \n",
       "L 92.042019 50.870421 \n",
       "L 92.377154 49.068951 \n",
       "L 92.712289 49.700972 \n",
       "L 93.047424 50.993229 \n",
       "L 93.382559 38.31818 \n",
       "L 93.717694 50.721253 \n",
       "L 94.052829 40.347659 \n",
       "L 94.387965 42.689784 \n",
       "L 94.7231 43.324578 \n",
       "L 95.058235 45.663703 \n",
       "L 95.39337 42.239311 \n",
       "L 95.728505 45.936374 \n",
       "L 96.06364 43.569031 \n",
       "L 96.398775 47.258761 \n",
       "L 96.73391 48.574307 \n",
       "L 97.069046 41.425352 \n",
       "L 97.404181 48.303827 \n",
       "L 97.739316 42.82277 \n",
       "L 98.074451 40.842145 \n",
       "L 98.409586 39.857947 \n",
       "L 98.744721 41.581257 \n",
       "L 99.079856 37.558466 \n",
       "L 99.414992 39.158338 \n",
       "L 99.750127 34.971847 \n",
       "L 100.085262 39.395832 \n",
       "L 100.420397 35.402167 \n",
       "L 100.755532 35.454927 \n",
       "L 101.090667 45.49227 \n",
       "L 101.425802 41.370647 \n",
       "L 101.760938 44.639223 \n",
       "L 102.096073 45.995186 \n",
       "L 102.431208 46.604533 \n",
       "L 102.766343 40.720572 \n",
       "L 103.101478 51.742915 \n",
       "L 103.436613 44.665735 \n",
       "L 103.771748 47.554569 \n",
       "L 104.106883 42.97867 \n",
       "L 104.442019 48.585247 \n",
       "L 104.777154 45.821478 \n",
       "L 105.112289 50.733691 \n",
       "L 105.447424 51.172857 \n",
       "L 105.782559 52.080213 \n",
       "L 106.117694 46.375264 \n",
       "L 106.452829 44.473088 \n",
       "L 106.787965 47.582539 \n",
       "L 107.1231 37.375768 \n",
       "L 107.79337 45.419129 \n",
       "L 108.128505 41.408407 \n",
       "L 108.46364 42.803956 \n",
       "L 108.798775 47.474485 \n",
       "L 109.13391 38.904826 \n",
       "L 109.469046 43.174211 \n",
       "L 109.804181 38.993511 \n",
       "L 110.474451 45.197015 \n",
       "L 111.144721 43.490601 \n",
       "L 111.479856 44.192134 \n",
       "L 111.814992 42.863906 \n",
       "L 112.150127 44.527576 \n",
       "L 112.485262 49.533332 \n",
       "L 112.820397 41.401994 \n",
       "L 113.155532 44.924933 \n",
       "L 113.490667 45.693776 \n",
       "L 113.825802 37.853085 \n",
       "L 114.160938 45.178811 \n",
       "L 114.496073 43.968087 \n",
       "L 114.831208 46.796019 \n",
       "L 115.166343 44.313463 \n",
       "L 115.501478 43.146125 \n",
       "L 115.836613 40.386133 \n",
       "L 116.171748 42.219214 \n",
       "L 116.506883 37.078181 \n",
       "L 117.177154 43.791214 \n",
       "L 117.512289 42.113567 \n",
       "L 117.847424 42.948749 \n",
       "L 118.182559 40.803379 \n",
       "L 118.517694 44.481439 \n",
       "L 118.852829 40.351898 \n",
       "L 119.187965 44.895591 \n",
       "L 119.5231 45.075625 \n",
       "L 119.858235 43.388261 \n",
       "L 120.19337 44.461866 \n",
       "L 120.528505 44.394249 \n",
       "L 120.86364 45.312676 \n",
       "L 121.198775 41.208949 \n",
       "L 121.53391 43.144911 \n",
       "L 121.869046 41.122553 \n",
       "L 122.204181 40.478494 \n",
       "L 122.539316 47.374854 \n",
       "L 122.874451 48.602329 \n",
       "L 123.209586 50.363014 \n",
       "L 123.544721 47.336907 \n",
       "L 123.879856 47.259009 \n",
       "L 124.214992 54.23912 \n",
       "L 124.550127 49.956975 \n",
       "L 125.220397 61.807714 \n",
       "L 125.555532 54.746878 \n",
       "L 125.890667 58.944044 \n",
       "L 126.225802 57.415976 \n",
       "L 126.560938 60.007357 \n",
       "L 126.896073 56.862175 \n",
       "L 127.231208 51.275091 \n",
       "L 127.566343 59.947505 \n",
       "L 127.901478 52.056878 \n",
       "L 128.236613 58.28371 \n",
       "L 128.571748 51.681199 \n",
       "L 128.906883 62.857749 \n",
       "L 129.242019 54.125313 \n",
       "L 129.577154 60.26868 \n",
       "L 129.912289 59.648117 \n",
       "L 130.247424 56.664624 \n",
       "L 130.582559 61.605791 \n",
       "L 130.917694 45.764807 \n",
       "L 131.252829 49.57903 \n",
       "L 131.587965 45.621905 \n",
       "L 131.9231 47.956726 \n",
       "L 132.258235 56.312125 \n",
       "L 132.59337 59.919764 \n",
       "L 132.928505 59.519747 \n",
       "L 133.26364 60.5128 \n",
       "L 133.598775 50.725561 \n",
       "L 133.93391 47.53623 \n",
       "L 134.269046 49.892901 \n",
       "L 134.604181 57.404809 \n",
       "L 134.939316 54.836498 \n",
       "L 135.274451 63.0676 \n",
       "L 135.609586 68.059693 \n",
       "L 135.944721 60.833138 \n",
       "L 136.279856 67.933083 \n",
       "L 136.614992 62.573784 \n",
       "L 136.950127 66.226864 \n",
       "L 137.285262 63.318024 \n",
       "L 137.620397 64.005623 \n",
       "L 137.955532 71.473363 \n",
       "L 138.290667 69.128846 \n",
       "L 138.625802 74.208873 \n",
       "L 139.296073 65.128475 \n",
       "L 139.966343 68.857607 \n",
       "L 140.301478 67.719982 \n",
       "L 140.636613 70.745202 \n",
       "L 140.971748 68.307364 \n",
       "L 141.306883 63.00438 \n",
       "L 141.977154 71.511604 \n",
       "L 142.312289 70.807609 \n",
       "L 142.647424 72.821 \n",
       "L 142.982559 69.240271 \n",
       "L 143.317694 61.293951 \n",
       "L 143.652829 65.78174 \n",
       "L 143.987965 64.099235 \n",
       "L 144.3231 64.240307 \n",
       "L 144.658235 71.554509 \n",
       "L 144.99337 67.959653 \n",
       "L 145.998775 75.850044 \n",
       "L 146.33391 77.346353 \n",
       "L 146.669046 78.18929 \n",
       "L 147.004181 73.213242 \n",
       "L 148.009586 67.415153 \n",
       "L 148.344721 73.335079 \n",
       "L 148.679856 72.315899 \n",
       "L 149.014992 73.548104 \n",
       "L 149.350127 80.35939 \n",
       "L 149.685262 75.820942 \n",
       "L 150.020397 77.762745 \n",
       "L 150.355532 83.281653 \n",
       "L 150.690667 72.173386 \n",
       "L 151.025802 83.713635 \n",
       "L 151.360938 82.283334 \n",
       "L 151.696073 80.092185 \n",
       "L 152.031208 83.580473 \n",
       "L 152.366343 78.144922 \n",
       "L 152.701478 79.687453 \n",
       "L 153.036613 80.074949 \n",
       "L 153.371748 84.012471 \n",
       "L 153.706883 84.367428 \n",
       "L 154.042019 85.581677 \n",
       "L 154.377154 91.505134 \n",
       "L 154.712289 91.411126 \n",
       "L 155.047424 92.058232 \n",
       "L 155.382559 97.305922 \n",
       "L 155.717694 90.300646 \n",
       "L 156.052829 89.114337 \n",
       "L 156.387965 90.561748 \n",
       "L 156.7231 86.308028 \n",
       "L 157.39337 90.521554 \n",
       "L 157.728505 92.570986 \n",
       "L 158.06364 91.661967 \n",
       "L 158.398775 90.295983 \n",
       "L 158.73391 97.185537 \n",
       "L 159.069046 95.083295 \n",
       "L 159.404181 96.08862 \n",
       "L 159.739316 99.93637 \n",
       "L 160.074451 101.797226 \n",
       "L 160.409586 96.59689 \n",
       "L 160.744721 99.749398 \n",
       "L 161.079856 99.641049 \n",
       "L 161.414992 98.857943 \n",
       "L 161.750127 101.995637 \n",
       "L 162.085262 102.170441 \n",
       "L 162.420397 98.872142 \n",
       "L 163.090667 95.990263 \n",
       "L 163.425802 94.191057 \n",
       "L 163.760938 98.005774 \n",
       "L 164.096073 96.538976 \n",
       "L 164.431208 97.251307 \n",
       "L 164.766343 97.658146 \n",
       "L 165.436613 94.891789 \n",
       "L 165.771748 98.539998 \n",
       "L 166.106883 93.831074 \n",
       "L 166.777154 106.676086 \n",
       "L 167.112289 101.310513 \n",
       "L 167.447424 112.968445 \n",
       "L 167.782559 112.087345 \n",
       "L 168.117694 104.584158 \n",
       "L 168.452829 105.380145 \n",
       "L 168.787965 100.541676 \n",
       "L 169.1231 105.758742 \n",
       "L 169.458235 101.881219 \n",
       "L 169.79337 105.43507 \n",
       "L 170.128505 112.031694 \n",
       "L 170.46364 103.63969 \n",
       "L 170.798775 108.129087 \n",
       "L 171.13391 110.454092 \n",
       "L 171.469046 105.472902 \n",
       "L 171.804181 106.907522 \n",
       "L 172.139316 114.678812 \n",
       "L 172.474451 111.366848 \n",
       "L 172.809586 114.371665 \n",
       "L 173.144721 115.776584 \n",
       "L 173.479856 114.25886 \n",
       "L 173.814992 110.053906 \n",
       "L 174.150127 115.847217 \n",
       "L 174.485262 114.334196 \n",
       "L 174.820397 120.354165 \n",
       "L 175.490667 111.774824 \n",
       "L 176.160938 108.517024 \n",
       "L 176.496073 109.659987 \n",
       "L 177.166343 123.214801 \n",
       "L 177.836613 110.835846 \n",
       "L 178.171748 120.527412 \n",
       "L 178.506883 116.186217 \n",
       "L 179.177154 129.361676 \n",
       "L 179.512289 125.562381 \n",
       "L 179.847424 127.414547 \n",
       "L 180.182559 123.631967 \n",
       "L 180.517694 121.550788 \n",
       "L 181.187965 120.737442 \n",
       "L 181.5231 122.400491 \n",
       "L 181.858235 121.879697 \n",
       "L 182.19337 126.101647 \n",
       "L 182.528505 124.823597 \n",
       "L 182.86364 120.188642 \n",
       "L 183.198775 122.472103 \n",
       "L 183.53391 122.320548 \n",
       "L 183.869046 121.666116 \n",
       "L 184.204181 124.17173 \n",
       "L 184.539316 124.634188 \n",
       "L 184.874451 126.736478 \n",
       "L 185.209586 127.422818 \n",
       "L 185.544721 124.940012 \n",
       "L 185.879856 132.750966 \n",
       "L 186.214992 127.059149 \n",
       "L 186.550127 125.227571 \n",
       "L 186.885262 127.921439 \n",
       "L 187.220397 118.82652 \n",
       "L 187.555532 126.53725 \n",
       "L 187.890667 127.069024 \n",
       "L 188.225802 128.05787 \n",
       "L 188.560938 127.156148 \n",
       "L 188.896073 128.594033 \n",
       "L 189.231208 125.675678 \n",
       "L 189.566343 132.38543 \n",
       "L 189.901478 131.746303 \n",
       "L 190.571748 133.636545 \n",
       "L 190.906883 130.821952 \n",
       "L 191.242019 133.923432 \n",
       "L 191.577154 135.439556 \n",
       "L 191.912289 135.176283 \n",
       "L 192.247424 136.939927 \n",
       "L 192.582559 133.03667 \n",
       "L 192.917694 133.633318 \n",
       "L 193.252829 132.344064 \n",
       "L 193.587965 133.465805 \n",
       "L 193.9231 133.61585 \n",
       "L 194.258235 133.292527 \n",
       "L 194.59337 133.254583 \n",
       "L 194.928505 132.188624 \n",
       "L 195.26364 135.844339 \n",
       "L 195.598775 132.577172 \n",
       "L 195.93391 131.515458 \n",
       "L 196.269046 133.886793 \n",
       "L 196.604181 123.629827 \n",
       "L 197.274451 132.371793 \n",
       "L 197.609586 131.056144 \n",
       "L 197.944721 132.56541 \n",
       "L 198.279856 132.413987 \n",
       "L 198.614992 132.932032 \n",
       "L 198.950127 131.837571 \n",
       "L 199.285262 137.593442 \n",
       "L 199.620397 137.431739 \n",
       "L 199.955532 137.994417 \n",
       "L 200.290667 134.38249 \n",
       "L 200.625802 135.872626 \n",
       "L 200.960938 132.217989 \n",
       "L 201.296073 132.797796 \n",
       "L 201.631208 137.490121 \n",
       "L 201.966343 134.674119 \n",
       "L 202.301478 135.939898 \n",
       "L 202.636613 135.783146 \n",
       "L 202.971748 134.731026 \n",
       "L 203.306883 134.182538 \n",
       "L 203.642019 136.120093 \n",
       "L 203.977154 133.82114 \n",
       "L 204.647424 134.297963 \n",
       "L 204.982559 128.739278 \n",
       "L 205.317694 130.893575 \n",
       "L 205.652829 131.656565 \n",
       "L 205.987965 132.073967 \n",
       "L 206.3231 130.9265 \n",
       "L 206.658235 136.266209 \n",
       "L 206.99337 130.450224 \n",
       "L 207.328505 136.72867 \n",
       "L 207.998775 131.82662 \n",
       "L 208.33391 132.648994 \n",
       "L 208.669046 130.72218 \n",
       "L 209.339316 136.077246 \n",
       "L 209.674451 136.622292 \n",
       "L 210.009586 136.947675 \n",
       "L 210.344721 135.886125 \n",
       "L 210.679856 136.867844 \n",
       "L 211.014992 136.691284 \n",
       "L 211.350127 137.193074 \n",
       "L 211.685262 135.891779 \n",
       "L 212.020397 136.155529 \n",
       "L 212.355532 133.277003 \n",
       "L 212.690667 133.668611 \n",
       "L 213.025802 136.775739 \n",
       "L 213.360938 135.186125 \n",
       "L 213.696073 135.195747 \n",
       "L 214.031208 139.267659 \n",
       "L 214.366343 136.877932 \n",
       "L 214.701478 137.902752 \n",
       "L 215.706883 131.424092 \n",
       "L 216.042019 130.540204 \n",
       "L 216.377154 135.217865 \n",
       "L 216.712289 136.521068 \n",
       "L 217.047424 132.233582 \n",
       "L 217.382559 136.298574 \n",
       "L 218.052829 132.546536 \n",
       "L 218.387965 135.692238 \n",
       "L 218.7231 131.978928 \n",
       "L 219.058235 134.527997 \n",
       "L 219.39337 134.988168 \n",
       "L 219.728505 131.749823 \n",
       "L 220.06364 134.256809 \n",
       "L 220.398775 132.943573 \n",
       "L 220.73391 131.179765 \n",
       "L 221.069046 130.710777 \n",
       "L 221.404181 133.122883 \n",
       "L 221.739316 130.567936 \n",
       "L 222.074451 131.193365 \n",
       "L 222.409586 133.498271 \n",
       "L 222.744721 134.225491 \n",
       "L 223.079856 133.295739 \n",
       "L 223.414992 131.727587 \n",
       "L 223.750127 131.260882 \n",
       "L 224.085262 127.895424 \n",
       "L 224.755532 135.714074 \n",
       "L 225.090667 132.540899 \n",
       "L 225.425802 136.081585 \n",
       "L 225.760938 137.582405 \n",
       "L 226.096073 132.484153 \n",
       "L 226.431208 137.239586 \n",
       "L 227.101478 127.592904 \n",
       "L 227.436613 128.851255 \n",
       "L 227.771748 125.283616 \n",
       "L 228.106883 127.989148 \n",
       "L 228.442019 132.135136 \n",
       "L 228.777154 131.196097 \n",
       "L 229.112289 132.306002 \n",
       "L 229.447424 134.12165 \n",
       "L 229.782559 134.011234 \n",
       "L 230.117694 135.550058 \n",
       "L 230.452829 134.146446 \n",
       "L 231.1231 128.141572 \n",
       "L 231.458235 127.519124 \n",
       "L 231.79337 126.133854 \n",
       "L 232.128505 130.15423 \n",
       "L 232.46364 131.179973 \n",
       "L 232.798775 133.563605 \n",
       "L 233.13391 132.84557 \n",
       "L 233.469046 125.161933 \n",
       "L 233.804181 127.262817 \n",
       "L 234.139316 123.274806 \n",
       "L 234.474451 124.541283 \n",
       "L 234.809586 127.073666 \n",
       "L 235.144721 122.858695 \n",
       "L 235.479856 126.111192 \n",
       "L 235.814992 119.618726 \n",
       "L 236.150127 117.489011 \n",
       "L 236.485262 122.107758 \n",
       "L 236.820397 119.141158 \n",
       "L 237.155532 118.735407 \n",
       "L 237.490667 119.07091 \n",
       "L 237.825802 123.000653 \n",
       "L 238.160938 123.064788 \n",
       "L 238.496073 122.1217 \n",
       "L 238.831208 127.036726 \n",
       "L 239.166343 123.90322 \n",
       "L 239.501478 124.757501 \n",
       "L 239.836613 128.267836 \n",
       "L 240.171748 125.519876 \n",
       "L 240.506883 125.970166 \n",
       "L 240.842019 121.212861 \n",
       "L 241.177154 121.861418 \n",
       "L 241.512289 123.022892 \n",
       "L 241.847424 119.055426 \n",
       "L 242.182559 119.260615 \n",
       "L 242.517694 113.110381 \n",
       "L 242.852829 111.136567 \n",
       "L 243.187965 108.355792 \n",
       "L 243.5231 112.810782 \n",
       "L 243.858235 112.478631 \n",
       "L 244.19337 114.115478 \n",
       "L 244.528505 117.082607 \n",
       "L 244.86364 112.562695 \n",
       "L 245.198775 120.618574 \n",
       "L 245.53391 115.868829 \n",
       "L 246.204181 121.412505 \n",
       "L 246.539316 111.489826 \n",
       "L 246.874451 112.153458 \n",
       "L 247.209586 104.407756 \n",
       "L 247.544721 103.964284 \n",
       "L 247.879856 107.116451 \n",
       "L 248.214992 102.10554 \n",
       "L 248.550127 99.668048 \n",
       "L 248.885262 101.142039 \n",
       "L 249.220397 101.153315 \n",
       "L 249.555532 105.571296 \n",
       "L 249.890667 112.783535 \n",
       "L 250.225802 115.56006 \n",
       "L 250.560938 115.110594 \n",
       "L 250.896073 117.237984 \n",
       "L 251.231208 115.256014 \n",
       "L 251.566343 119.749565 \n",
       "L 251.901478 114.347442 \n",
       "L 252.236613 114.539509 \n",
       "L 252.906883 106.60493 \n",
       "L 253.577154 113.373778 \n",
       "L 253.912289 113.433895 \n",
       "L 254.247424 109.771141 \n",
       "L 254.582559 102.121331 \n",
       "L 254.917694 97.178653 \n",
       "L 255.252829 100.485496 \n",
       "L 255.587965 100.190699 \n",
       "L 255.9231 103.352254 \n",
       "L 256.258235 101.752778 \n",
       "L 256.59337 93.162045 \n",
       "L 256.928505 94.914327 \n",
       "L 257.26364 95.431748 \n",
       "L 257.93391 97.271222 \n",
       "L 258.269046 93.93215 \n",
       "L 258.604181 89.221019 \n",
       "L 258.939316 92.300361 \n",
       "L 259.274451 92.338772 \n",
       "L 259.609586 89.137857 \n",
       "L 259.944721 94.914521 \n",
       "L 260.279856 94.501921 \n",
       "L 260.614992 85.894493 \n",
       "L 260.950127 96.110886 \n",
       "L 261.285262 92.722657 \n",
       "L 261.620397 86.202652 \n",
       "L 261.955532 93.747523 \n",
       "L 262.625802 84.562271 \n",
       "L 262.960938 91.820956 \n",
       "L 263.296073 88.81449 \n",
       "L 263.631208 91.769565 \n",
       "L 263.966343 91.629339 \n",
       "L 264.301478 91.289313 \n",
       "L 264.636613 89.899971 \n",
       "L 264.971748 83.487627 \n",
       "L 265.306883 84.967603 \n",
       "L 265.642019 85.704732 \n",
       "L 265.977154 85.736763 \n",
       "L 266.312289 91.976362 \n",
       "L 266.647424 94.582994 \n",
       "L 266.982559 88.511408 \n",
       "L 267.317694 89.79989 \n",
       "L 267.652829 89.240852 \n",
       "L 267.987965 85.793626 \n",
       "L 268.3231 89.191184 \n",
       "L 268.658235 84.103499 \n",
       "L 269.328505 78.377801 \n",
       "L 269.66364 79.561709 \n",
       "L 269.998775 85.184225 \n",
       "L 270.33391 77.373963 \n",
       "L 270.669046 82.737408 \n",
       "L 271.339316 67.389935 \n",
       "L 271.674451 73.397248 \n",
       "L 272.009586 68.826676 \n",
       "L 272.679856 73.898575 \n",
       "L 273.014992 80.915908 \n",
       "L 273.350127 79.872724 \n",
       "L 274.020397 83.736527 \n",
       "L 274.355532 72.061451 \n",
       "L 274.690667 71.726296 \n",
       "L 275.025802 75.190677 \n",
       "L 275.360937 71.818967 \n",
       "L 275.696073 74.958296 \n",
       "L 276.031208 75.173071 \n",
       "L 276.366343 72.954194 \n",
       "L 276.701478 72.124862 \n",
       "L 277.036613 69.679051 \n",
       "L 277.371748 70.736866 \n",
       "L 277.706883 66.370278 \n",
       "L 278.042019 66.953622 \n",
       "L 278.377154 70.768538 \n",
       "L 278.712289 64.660079 \n",
       "L 279.047424 70.753191 \n",
       "L 279.382559 63.186258 \n",
       "L 279.717694 63.747029 \n",
       "L 280.052829 62.060867 \n",
       "L 280.387965 54.030367 \n",
       "L 280.7231 62.457597 \n",
       "L 281.058235 57.894216 \n",
       "L 281.728505 63.532163 \n",
       "L 282.06364 57.80658 \n",
       "L 282.398775 64.655618 \n",
       "L 282.73391 58.839236 \n",
       "L 283.069046 62.106495 \n",
       "L 283.404181 68.097648 \n",
       "L 283.739316 62.786071 \n",
       "L 284.074451 65.851315 \n",
       "L 284.409586 65.782654 \n",
       "L 284.744721 67.370647 \n",
       "L 285.079856 67.658669 \n",
       "L 285.414992 62.838066 \n",
       "L 285.750127 65.554202 \n",
       "L 286.085262 63.045505 \n",
       "L 286.420397 54.594003 \n",
       "L 286.755532 54.77293 \n",
       "L 287.090667 53.591451 \n",
       "L 287.425802 51.209815 \n",
       "L 287.760938 53.539945 \n",
       "L 288.096073 51.741477 \n",
       "L 288.431208 59.589435 \n",
       "L 288.766343 58.136403 \n",
       "L 289.101478 54.769516 \n",
       "L 289.436613 59.630335 \n",
       "L 289.771748 45.043738 \n",
       "L 290.106883 48.634864 \n",
       "L 290.442019 46.063878 \n",
       "L 290.777154 47.899881 \n",
       "L 291.112289 51.109517 \n",
       "L 291.447424 60.36404 \n",
       "L 291.782559 52.985793 \n",
       "L 292.117694 53.989162 \n",
       "L 292.787965 46.269841 \n",
       "L 293.1231 47.129736 \n",
       "L 293.79337 56.480319 \n",
       "L 294.128505 53.760937 \n",
       "L 294.46364 61.580639 \n",
       "L 294.798775 53.510424 \n",
       "L 295.13391 48.715106 \n",
       "L 295.469046 47.967826 \n",
       "L 295.804181 53.416832 \n",
       "L 296.139316 51.998786 \n",
       "L 296.474451 58.460828 \n",
       "L 296.809586 61.643425 \n",
       "L 297.479856 49.962663 \n",
       "L 297.814992 50.339699 \n",
       "L 298.150127 45.288674 \n",
       "L 298.485262 37.82684 \n",
       "L 298.820397 44.820416 \n",
       "L 299.155532 46.127613 \n",
       "L 299.490667 42.20105 \n",
       "L 299.825802 46.725796 \n",
       "L 300.160938 47.355824 \n",
       "L 300.496073 48.405368 \n",
       "L 300.831208 44.186233 \n",
       "L 301.501478 47.296438 \n",
       "L 301.836613 47.33831 \n",
       "L 302.171748 45.48919 \n",
       "L 302.506883 52.365923 \n",
       "L 302.842019 52.048739 \n",
       "L 303.177154 46.326245 \n",
       "L 303.512289 51.420934 \n",
       "L 304.182559 46.173963 \n",
       "L 304.517694 45.38501 \n",
       "L 304.852829 46.621288 \n",
       "L 305.5231 41.314067 \n",
       "L 305.858235 41.839751 \n",
       "L 306.19337 43.088614 \n",
       "L 306.528505 40.440791 \n",
       "L 306.86364 43.498141 \n",
       "L 307.198775 42.322716 \n",
       "L 307.53391 46.498843 \n",
       "L 307.869046 49.198304 \n",
       "L 308.204181 40.311004 \n",
       "L 308.539316 50.266767 \n",
       "L 309.209586 40.430168 \n",
       "L 309.544721 49.846243 \n",
       "L 310.214992 42.235865 \n",
       "L 310.550127 51.18428 \n",
       "L 311.220397 43.601775 \n",
       "L 311.555532 45.281764 \n",
       "L 311.890667 44.611816 \n",
       "L 312.560938 39.358434 \n",
       "L 312.896073 40.204511 \n",
       "L 313.231208 41.906758 \n",
       "L 313.566343 37.424554 \n",
       "L 313.901478 44.424135 \n",
       "L 314.236613 43.363384 \n",
       "L 314.571748 44.479903 \n",
       "L 314.906883 41.339703 \n",
       "L 315.242019 43.819383 \n",
       "L 315.577154 40.156668 \n",
       "L 315.912289 34.900543 \n",
       "L 316.582559 39.153409 \n",
       "L 316.917694 31.483694 \n",
       "L 317.252829 38.010326 \n",
       "L 317.587965 37.160275 \n",
       "L 317.9231 42.685741 \n",
       "L 318.258235 37.325901 \n",
       "L 318.59337 45.922412 \n",
       "L 318.928505 47.646985 \n",
       "L 319.26364 41.145623 \n",
       "L 319.598775 42.050292 \n",
       "L 319.93391 45.271799 \n",
       "L 320.269046 40.861632 \n",
       "L 320.939316 48.086844 \n",
       "L 321.274451 44.80365 \n",
       "L 321.609586 47.628286 \n",
       "L 321.944721 48.664867 \n",
       "L 322.279856 43.843902 \n",
       "L 322.614992 44.351882 \n",
       "L 322.950127 46.00731 \n",
       "L 323.285262 40.601477 \n",
       "L 323.620397 41.11266 \n",
       "L 323.955532 44.193727 \n",
       "L 324.290667 37.748445 \n",
       "L 324.625802 46.825428 \n",
       "L 324.960938 39.812279 \n",
       "L 325.296073 43.549211 \n",
       "L 325.631208 45.560063 \n",
       "L 325.966343 43.017413 \n",
       "L 326.301478 39.110053 \n",
       "L 326.636613 49.057079 \n",
       "L 326.971748 44.701301 \n",
       "L 327.306883 47.811064 \n",
       "L 327.642019 49.115747 \n",
       "L 327.977154 44.717745 \n",
       "L 328.312289 44.123099 \n",
       "L 328.647424 40.751798 \n",
       "L 328.982559 45.649692 \n",
       "L 329.317694 45.587677 \n",
       "L 329.652829 39.866799 \n",
       "L 329.987965 50.760894 \n",
       "L 330.658235 45.706467 \n",
       "L 330.99337 52.113196 \n",
       "L 331.328505 44.343476 \n",
       "L 331.66364 50.537518 \n",
       "L 331.998775 49.374367 \n",
       "L 332.33391 46.318978 \n",
       "L 332.669046 48.771867 \n",
       "L 333.004181 52.162402 \n",
       "L 333.339316 44.657551 \n",
       "L 333.674451 52.799941 \n",
       "L 334.009586 54.810457 \n",
       "L 334.344721 53.140258 \n",
       "L 334.679856 53.908451 \n",
       "L 335.014992 49.872957 \n",
       "L 335.350127 53.737942 \n",
       "L 335.685262 50.805918 \n",
       "L 336.020397 51.276684 \n",
       "L 336.355532 53.871059 \n",
       "L 336.690667 50.388914 \n",
       "L 337.025802 56.954913 \n",
       "L 337.360937 54.661376 \n",
       "L 337.696073 53.284595 \n",
       "L 338.031208 55.472651 \n",
       "L 338.366343 51.189836 \n",
       "L 338.701478 52.5415 \n",
       "L 339.036613 48.304618 \n",
       "L 339.371748 51.114201 \n",
       "L 339.706883 48.519999 \n",
       "L 340.042019 54.259173 \n",
       "L 340.377154 49.032059 \n",
       "L 340.712289 60.064661 \n",
       "L 341.047424 57.338523 \n",
       "L 341.382559 49.734297 \n",
       "L 341.717694 63.67551 \n",
       "L 342.052829 50.648698 \n",
       "L 342.387965 54.336418 \n",
       "L 342.7231 51.793208 \n",
       "L 343.39337 54.330868 \n",
       "L 343.728505 51.624919 \n",
       "L 344.06364 58.171992 \n",
       "L 344.398775 58.25622 \n",
       "L 344.73391 54.638248 \n",
       "L 345.069046 59.344976 \n",
       "L 345.404181 61.337322 \n",
       "L 345.739316 64.864031 \n",
       "L 346.074451 66.222985 \n",
       "L 346.409586 59.223048 \n",
       "L 346.744721 64.368139 \n",
       "L 347.079856 57.719275 \n",
       "L 347.750127 67.555676 \n",
       "L 348.085262 70.689852 \n",
       "L 348.420397 67.374627 \n",
       "L 348.755532 61.663153 \n",
       "L 349.090667 62.351886 \n",
       "L 349.425802 60.386084 \n",
       "L 349.760938 62.865076 \n",
       "L 350.096073 67.58333 \n",
       "L 350.431208 65.675186 \n",
       "L 350.766343 68.210583 \n",
       "L 351.101478 65.738622 \n",
       "L 351.436613 65.663323 \n",
       "L 351.771748 62.392085 \n",
       "L 352.106883 63.109599 \n",
       "L 352.442019 58.586181 \n",
       "L 352.777154 60.252828 \n",
       "L 353.112289 61.284587 \n",
       "L 353.447424 61.143337 \n",
       "L 354.117694 70.32701 \n",
       "L 354.452829 72.593206 \n",
       "L 354.787965 75.884219 \n",
       "L 355.1231 80.311714 \n",
       "L 355.458235 76.853229 \n",
       "L 355.79337 75.778105 \n",
       "L 356.128505 74.293009 \n",
       "L 356.46364 68.94895 \n",
       "L 356.798775 77.263937 \n",
       "L 357.13391 77.081573 \n",
       "L 357.804181 83.378371 \n",
       "L 358.139316 79.959651 \n",
       "L 358.474451 79.514621 \n",
       "L 358.809586 80.295071 \n",
       "L 359.144721 78.074782 \n",
       "L 359.479856 77.804029 \n",
       "L 359.814992 80.644633 \n",
       "L 360.150127 78.517888 \n",
       "L 360.485262 82.787074 \n",
       "L 360.820397 82.171267 \n",
       "L 361.155532 80.214513 \n",
       "L 361.490667 81.876902 \n",
       "L 361.825802 77.378403 \n",
       "L 362.160938 76.725256 \n",
       "L 362.496073 81.72061 \n",
       "L 362.831208 76.929866 \n",
       "L 363.501478 84.541794 \n",
       "L 363.836613 83.676616 \n",
       "L 364.171748 89.780455 \n",
       "L 364.506883 84.621865 \n",
       "L 364.842019 85.162209 \n",
       "L 365.177154 84.515318 \n",
       "L 365.512289 83.489087 \n",
       "L 365.847424 89.34879 \n",
       "L 366.182559 88.25261 \n",
       "L 366.517694 93.621849 \n",
       "L 366.852829 93.752013 \n",
       "L 367.187965 92.198653 \n",
       "L 367.5231 95.143068 \n",
       "L 367.858235 96.797149 \n",
       "L 368.528505 85.005989 \n",
       "L 368.86364 90.899949 \n",
       "L 369.198775 92.306805 \n",
       "L 369.53391 88.512657 \n",
       "L 369.869046 99.035877 \n",
       "L 370.204181 94.312053 \n",
       "L 370.539316 98.242024 \n",
       "L 370.874451 93.878315 \n",
       "L 371.209586 92.844354 \n",
       "L 371.544721 96.789951 \n",
       "L 371.879856 94.199366 \n",
       "L 372.214992 89.847448 \n",
       "L 372.550127 92.51421 \n",
       "L 373.220397 85.955963 \n",
       "L 373.555532 89.642181 \n",
       "L 374.225802 102.401482 \n",
       "L 375.231208 110.518933 \n",
       "L 375.901478 107.424647 \n",
       "L 376.236613 113.142198 \n",
       "L 376.571748 112.869726 \n",
       "L 376.906883 107.30403 \n",
       "L 377.242019 111.715211 \n",
       "L 377.577154 114.218865 \n",
       "L 377.912289 113.82727 \n",
       "L 378.247424 112.730645 \n",
       "L 378.582559 115.005922 \n",
       "L 379.252829 101.677745 \n",
       "L 379.587965 109.764235 \n",
       "L 379.9231 104.768832 \n",
       "L 380.258235 116.50041 \n",
       "L 380.59337 121.633915 \n",
       "L 380.928505 116.093114 \n",
       "L 381.26364 124.032214 \n",
       "L 381.598775 111.429276 \n",
       "L 381.93391 115.405486 \n",
       "L 382.604181 118.205459 \n",
       "L 382.939316 122.821925 \n",
       "L 383.274451 121.390845 \n",
       "L 383.609586 120.900566 \n",
       "L 383.944721 114.392495 \n",
       "L 384.279856 115.718641 \n",
       "L 384.614992 113.535489 \n",
       "L 384.950127 113.463302 \n",
       "L 385.285262 117.738397 \n",
       "L 385.620397 118.650615 \n",
       "L 385.955532 116.175876 \n",
       "L 386.290667 115.911976 \n",
       "L 386.625802 114.908465 \n",
       "L 386.960938 112.29762 \n",
       "L 386.960938 112.29762 \n",
       "\" clip-path=\"url(#pe84e13ccd4)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 52.160938 173.52 \n",
       "L 52.160938 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 386.960938 173.52 \n",
       "L 386.960938 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 52.160938 173.52 \n",
       "L 386.960938 173.52 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 52.160938 7.2 \n",
       "L 386.960938 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 59.160938 168.52 \n",
       "L 153.885938 168.52 \n",
       "Q 155.885938 168.52 155.885938 166.52 \n",
       "L 155.885938 138.16375 \n",
       "Q 155.885938 136.16375 153.885938 136.16375 \n",
       "L 59.160938 136.16375 \n",
       "Q 57.160938 136.16375 57.160938 138.16375 \n",
       "L 57.160938 166.52 \n",
       "Q 57.160938 168.52 59.160938 168.52 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_27\">\n",
       "     <path d=\"M 61.160938 144.262188 \n",
       "L 71.160938 144.262188 \n",
       "L 81.160938 144.262188 \n",
       "\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    </g>\n",
       "    <g id=\"text_15\">\n",
       "     <!-- data -->\n",
       "     <g transform=\"translate(89.160938 147.762188)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \n",
       "L 2906 4863 \n",
       "L 3481 4863 \n",
       "L 3481 0 \n",
       "L 2906 0 \n",
       "L 2906 525 \n",
       "Q 2725 213 2448 61 \n",
       "Q 2172 -91 1784 -91 \n",
       "Q 1150 -91 751 415 \n",
       "Q 353 922 353 1747 \n",
       "Q 353 2572 751 3078 \n",
       "Q 1150 3584 1784 3584 \n",
       "Q 2172 3584 2448 3432 \n",
       "Q 2725 3281 2906 2969 \n",
       "z\n",
       "M 947 1747 \n",
       "Q 947 1113 1208 752 \n",
       "Q 1469 391 1925 391 \n",
       "Q 2381 391 2643 752 \n",
       "Q 2906 1113 2906 1747 \n",
       "Q 2906 2381 2643 2742 \n",
       "Q 2381 3103 1925 3103 \n",
       "Q 1469 3103 1208 2742 \n",
       "Q 947 2381 947 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-64\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"124.755859\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"163.964844\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_28\">\n",
       "     <path d=\"M 61.160938 158.940313 \n",
       "L 71.160938 158.940313 \n",
       "L 81.160938 158.940313 \n",
       "\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n",
       "    </g>\n",
       "    <g id=\"text_16\">\n",
       "     <!-- 1-step preds -->\n",
       "     <g transform=\"translate(89.160938 162.440313)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-2d\" d=\"M 313 2009 \n",
       "L 1997 2009 \n",
       "L 1997 1497 \n",
       "L 313 1497 \n",
       "L 313 2009 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-2d\" x=\"63.623047\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"99.707031\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"151.806641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"191.015625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"252.539062\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"316.015625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"347.802734\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"411.279297\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"450.142578\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-64\" x=\"511.666016\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"575.142578\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pe84e13ccd4\">\n",
       "   <rect x=\"52.160938\" y=\"7.2\" width=\"334.8\" height=\"166.32\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 600x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 通过模型生成一步预测\n",
    "onestep_preds = model(features)\n",
    "\n",
    "# 使用 d2l.plot 函数将原始数据和一步预测可视化为图表\n",
    "d2l.plot([time, time[tau:]],\n",
    "         [x.detach().numpy(), onestep_preds.detach().numpy()], 'time',\n",
    "         'x', legend=['data', '1-step preds'], xlim=[1, 1000],\n",
    "         figsize=(6, 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;从上图我们可以看到基于长短期记忆网络模型预测出来的结果在时间序列问题也能有良好的表现，能过很好地拟合所提供的序列数据。  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 9.3 深度循环神经网络 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.3.1 \n",
    "\n",
    "基于我们在8.5节中讨论的单层实现，尝试从零开始实现两层循环神经网络。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "模型结构：\n",
    "\n",
    "[输入层]\\\n",
    "[第一层RNN] --> (H1)\\\n",
    "[第二层RNN] --> (H2)\\\n",
    "[输出层]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# 定义训练数据\n",
    "import math\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch.nn import functional as F\n",
    "from d2l import torch as d2l\n",
    "\n",
    "batch_size, num_steps = 32, 35\n",
    "train_iter, vocab = d2l.load_data_time_machine(batch_size, num_steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_ch8(net, train_iter, vocab, lr, num_epochs, device,\n",
    "              use_random_iter=False):\n",
    "    \"\"\"\n",
    "    训练模型（定义见第8章）\n",
    "    \"\"\"\n",
    "    # 交叉熵损失函数\n",
    "    loss = nn.CrossEntropyLoss()\n",
    "\n",
    "    # 初始化优化器\n",
    "    if isinstance(net, nn.Module):\n",
    "        updater = torch.optim.SGD(net.parameters(), lr)\n",
    "    else:\n",
    "        updater = lambda batch_size: d2l.sgd(net.params, lr, batch_size)\n",
    "\n",
    "    # 预测函数\n",
    "    predict = lambda prefix: d2l.predict_ch8(prefix, 50, net, vocab, device)\n",
    "\n",
    "    # 训练和预测\n",
    "    for epoch in range(num_epochs):\n",
    "        # 调用train_epoch_ch8函数进行一轮训练\n",
    "        ppl, speed = d2l.train_epoch_ch8(net, train_iter, loss, updater, device, use_random_iter)\n",
    "        \n",
    "        # 每10轮打印一次困惑度\n",
    "        if (epoch + 1) % 10 == 0:\n",
    "            print(f'Epoch {epoch + 1}, Perplexity: {ppl:.1f}')\n",
    "\n",
    "    # 打印一个例子的预测结果\n",
    "    print(predict('time traveller'))\n",
    "\n",
    "    # 返回最终的困惑度值\n",
    "    return ppl\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# 初始化模型参数\n",
    "def get_params(vocab_size, num_hiddens, device):\n",
    "    num_inputs = num_outputs = vocab_size\n",
    "\n",
    "    def normal(shape):\n",
    "        return torch.randn(size=shape, device=device) * 0.01\n",
    "\n",
    "    # 第一层RNN参数\n",
    "    W_xh1 = normal((num_inputs, num_hiddens))\n",
    "    W_hh1 = normal((num_hiddens, num_hiddens))\n",
    "    b_h1 = torch.zeros(num_hiddens, device=device)\n",
    "    \n",
    "    # 第二层RNN参数\n",
    "    W_hh2 = normal((num_hiddens, num_hiddens))\n",
    "    b_h2 = torch.zeros(num_hiddens, device=device)\n",
    "\n",
    "    # 输出层参数\n",
    "    W_hq = normal((num_hiddens, num_outputs))\n",
    "    b_q = torch.zeros(num_outputs, device=device)\n",
    "\n",
    "    # 将参数放在一个列表中\n",
    "    params = [W_xh1, W_hh1, b_h1, W_hh2, b_h2, W_hq, b_q]\n",
    "\n",
    "    # 将参数的requires_grad属性设置为True，以便进行梯度计算\n",
    "    for param in params:\n",
    "        param.requires_grad_(True)\n",
    "\n",
    "    return params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def init_rnn_state(batch_size, num_hiddens, device):\n",
    "    # 返回初始化的双层RNN状态\n",
    "    return (torch.zeros((batch_size, num_hiddens), device=device),\n",
    "            torch.zeros((batch_size, num_hiddens), device=device))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;定义一个双层RNN网络"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def rnn(inputs, state, params):\n",
    "    # inputs的形状：(时间步数量，批量大小，词表大小)\n",
    "    W_xh1, W_hh1, b_h1, W_hh2, b_h2, W_hq, b_q = params\n",
    "    H1, H2 = state\n",
    "    outputs = []\n",
    "\n",
    "    # X的形状：(批量大小，词表大小)\n",
    "    for X in inputs:\n",
    "        # 第一层RNN的计算\n",
    "        H1 = torch.relu(torch.mm(X, W_xh1) + torch.mm(H1, W_hh1) + b_h1)\n",
    "        # 第二层RNN的计算\n",
    "        H2 = torch.relu(torch.mm(H1, W_hh2) + b_h2)\n",
    "        # 输出层的计算\n",
    "        Y = torch.mm(H2, W_hq) + b_q\n",
    "        outputs.append(Y)\n",
    "\n",
    "    # 将所有输出连接在一起\n",
    "    return torch.cat(outputs, dim=0), (H1, H2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "num_hiddens = 512\n",
    "net = d2l.RNNModelScratch(len(vocab), num_hiddens, d2l.try_gpu(), get_params,\n",
    "                      init_rnn_state, rnn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "perplexity 1.0, 51951.0 tokens/sec on cuda:0\n",
      "time travelleryou can show black is white by argument said filby\n",
      "travelleryou can show black is white by argument said filby\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"252.646875pt\" height=\"183.35625pt\" viewBox=\"0 0 252.646875 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2024-02-14T19:42:13.568427</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 183.35625 \n",
       "L 252.646875 183.35625 \n",
       "L 252.646875 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "L 40.603125 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 76.474554 145.8 \n",
       "L 76.474554 7.2 \n",
       "\" clip-path=\"url(#p1021cdf20e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"m31ee3f3207\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m31ee3f3207\" x=\"76.474554\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(66.930804 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 116.331696 145.8 \n",
       "L 116.331696 7.2 \n",
       "\" clip-path=\"url(#p1021cdf20e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m31ee3f3207\" x=\"116.331696\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 200 -->\n",
       "      <g transform=\"translate(106.787946 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 156.188839 145.8 \n",
       "L 156.188839 7.2 \n",
       "\" clip-path=\"url(#p1021cdf20e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m31ee3f3207\" x=\"156.188839\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 300 -->\n",
       "      <g transform=\"translate(146.645089 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 196.045982 145.8 \n",
       "L 196.045982 7.2 \n",
       "\" clip-path=\"url(#p1021cdf20e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m31ee3f3207\" x=\"196.045982\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 400 -->\n",
       "      <g transform=\"translate(186.502232 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "\" clip-path=\"url(#p1021cdf20e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m31ee3f3207\" x=\"235.903125\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 500 -->\n",
       "      <g transform=\"translate(226.359375 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(123.025 174.076563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 40.603125 109.001087 \n",
       "L 235.903125 109.001087 \n",
       "\" clip-path=\"url(#p1021cdf20e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"m2aa8eb547e\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m2aa8eb547e\" x=\"40.603125\" y=\"109.001087\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(27.240625 112.800305)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 40.603125 70.710515 \n",
       "L 235.903125 70.710515 \n",
       "\" clip-path=\"url(#p1021cdf20e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m2aa8eb547e\" x=\"40.603125\" y=\"70.710515\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(20.878125 74.509734)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 40.603125 32.419943 \n",
       "L 235.903125 32.419943 \n",
       "\" clip-path=\"url(#p1021cdf20e)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m2aa8eb547e\" x=\"40.603125\" y=\"32.419943\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 15 -->\n",
       "      <g transform=\"translate(20.878125 36.219162)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_10\">\n",
       "     <!-- perplexity -->\n",
       "     <g transform=\"translate(14.798437 101.626563)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"229.589844\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"257.373047\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"317.146484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"376.326172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"404.109375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"443.318359\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_17\">\n",
       "    <path d=\"M 40.603125 13.5 \n",
       "L 44.588839 32.675445 \n",
       "L 48.574554 58.139584 \n",
       "L 52.560268 67.18703 \n",
       "L 56.545982 72.453717 \n",
       "L 60.531696 82.182787 \n",
       "L 64.517411 86.509042 \n",
       "L 68.503125 96.008868 \n",
       "L 72.488839 100.973876 \n",
       "L 76.474554 107.630089 \n",
       "L 80.460268 112.432131 \n",
       "L 84.445982 118.88408 \n",
       "L 88.431696 122.703212 \n",
       "L 92.417411 126.655997 \n",
       "L 96.403125 130.557152 \n",
       "L 100.388839 133.495136 \n",
       "L 104.374554 135.377086 \n",
       "L 108.360268 136.889649 \n",
       "L 112.345982 137.931392 \n",
       "L 116.331696 138.692916 \n",
       "L 120.317411 138.800666 \n",
       "L 124.303125 139.104233 \n",
       "L 128.288839 138.768433 \n",
       "L 132.274554 139.093808 \n",
       "L 136.260268 139.256072 \n",
       "L 140.245982 139.303587 \n",
       "L 144.231696 139.402697 \n",
       "L 148.217411 138.608615 \n",
       "L 152.203125 139.353893 \n",
       "L 156.188839 139.310479 \n",
       "L 160.174554 139.359622 \n",
       "L 164.160268 139.337333 \n",
       "L 168.145982 139.385644 \n",
       "L 172.131696 139.394198 \n",
       "L 176.117411 139.453231 \n",
       "L 180.103125 139.340986 \n",
       "L 184.088839 139.4543 \n",
       "L 188.074554 139.449729 \n",
       "L 192.060268 139.424412 \n",
       "L 196.045982 139.462858 \n",
       "L 200.031696 139.439624 \n",
       "L 204.017411 139.46034 \n",
       "L 208.003125 139.467278 \n",
       "L 211.988839 139.468685 \n",
       "L 215.974554 139.447342 \n",
       "L 219.960268 139.435746 \n",
       "L 223.945982 139.446044 \n",
       "L 227.931696 139.5 \n",
       "L 231.917411 139.439874 \n",
       "L 235.903125 139.429364 \n",
       "\" clip-path=\"url(#p1021cdf20e)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 40.603125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 235.903125 145.8 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 40.603125 7.2 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 173.628125 29.878125 \n",
       "L 228.903125 29.878125 \n",
       "Q 230.903125 29.878125 230.903125 27.878125 \n",
       "L 230.903125 14.2 \n",
       "Q 230.903125 12.2 228.903125 12.2 \n",
       "L 173.628125 12.2 \n",
       "Q 171.628125 12.2 171.628125 14.2 \n",
       "L 171.628125 27.878125 \n",
       "Q 171.628125 29.878125 173.628125 29.878125 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_18\">\n",
       "     <path d=\"M 175.628125 20.298438 \n",
       "L 185.628125 20.298438 \n",
       "L 195.628125 20.298438 \n",
       "\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    </g>\n",
       "    <g id=\"text_11\">\n",
       "     <!-- train -->\n",
       "     <g transform=\"translate(203.628125 23.798438)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p1021cdf20e\">\n",
       "   <rect x=\"40.603125\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 350x250 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "num_epochs, lr = 500, 1\n",
    "d2l.train_ch8(net, train_iter, vocab, lr, num_epochs, d2l.try_gpu())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.3.2\n",
    "\n",
    "在本节训练模型中，比较使用门控循环单元替换长短期记忆网络后模型的精确度和训练速度。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from d2l import torch as d2l\n",
    "\n",
    "batch_size, num_steps = 32, 35\n",
    "train_iter, vocab = d2l.load_data_time_machine(batch_size, num_steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_ch8(net, train_iter, vocab, lr, num_epochs, device,\n",
    "              use_random_iter=False):\n",
    "    \"\"\"\n",
    "    训练模型（定义见第8章）\n",
    "    \"\"\"\n",
    "    # 交叉熵损失函数\n",
    "    loss = nn.CrossEntropyLoss()\n",
    "\n",
    "    # 初始化优化器\n",
    "    if isinstance(net, nn.Module):\n",
    "        updater = torch.optim.SGD(net.parameters(), lr)\n",
    "    else:\n",
    "        updater = lambda batch_size: d2l.sgd(net.params, lr, batch_size)\n",
    "\n",
    "    # 预测函数\n",
    "    predict = lambda prefix: d2l.predict_ch8(prefix, 50, net, vocab, device)\n",
    "\n",
    "    # 训练和预测\n",
    "    for epoch in range(num_epochs):\n",
    "        # 调用train_epoch_ch8函数进行一轮训练\n",
    "        ppl, speed = d2l.train_epoch_ch8(net, train_iter, loss, updater, device, use_random_iter)\n",
    "        \n",
    "        # 每10轮打印一次困惑度\n",
    "        if (epoch + 1) % 10 == 0:\n",
    "            print(f'Epoch {epoch + 1}, Perplexity: {ppl:.1f}')\n",
    "\n",
    "    # 打印一个例子的预测结果\n",
    "    print(predict('time traveller'))\n",
    "\n",
    "    # 返回最终的困惑度值\n",
    "    return ppl\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "vocab_size, num_hiddens, num_layers = len(vocab), 256, 2\n",
    "num_inputs = vocab_size\n",
    "device = d2l.try_gpu()\n",
    "\n",
    "# 定义LSTM层\n",
    "lstm_layer = nn.LSTM(num_inputs, num_hiddens, num_layers)\n",
    "\n",
    "# 创建RNN模型\n",
    "model = d2l.RNNModel(lstm_layer, len(vocab))\n",
    "\n",
    "# 将模型移动到GPU（如果可用）\n",
    "model = model.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "perplexity 1.0, 162486.3 tokens/sec on cuda:0\n",
      "time travelleryou can show black is white by argument said filby\n",
      "travelleryou can show black is white by argument said filby\n",
      "LSTM_Timer: 34.48\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"252.646875pt\" height=\"183.35625pt\" viewBox=\"0 0 252.646875 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2024-02-14T19:42:48.310139</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 183.35625 \n",
       "L 252.646875 183.35625 \n",
       "L 252.646875 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "L 40.603125 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 76.474554 145.8 \n",
       "L 76.474554 7.2 \n",
       "\" clip-path=\"url(#pd5a62a2a66)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"maf061ff14c\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#maf061ff14c\" x=\"76.474554\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(66.930804 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 116.331696 145.8 \n",
       "L 116.331696 7.2 \n",
       "\" clip-path=\"url(#pd5a62a2a66)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#maf061ff14c\" x=\"116.331696\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 200 -->\n",
       "      <g transform=\"translate(106.787946 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 156.188839 145.8 \n",
       "L 156.188839 7.2 \n",
       "\" clip-path=\"url(#pd5a62a2a66)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#maf061ff14c\" x=\"156.188839\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 300 -->\n",
       "      <g transform=\"translate(146.645089 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 196.045982 145.8 \n",
       "L 196.045982 7.2 \n",
       "\" clip-path=\"url(#pd5a62a2a66)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#maf061ff14c\" x=\"196.045982\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 400 -->\n",
       "      <g transform=\"translate(186.502232 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "\" clip-path=\"url(#pd5a62a2a66)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#maf061ff14c\" x=\"235.903125\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 500 -->\n",
       "      <g transform=\"translate(226.359375 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(123.025 174.076563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 40.603125 109.144968 \n",
       "L 235.903125 109.144968 \n",
       "\" clip-path=\"url(#pd5a62a2a66)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"m5be09a0eaa\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5be09a0eaa\" x=\"40.603125\" y=\"109.144968\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(27.240625 112.944186)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 40.603125 70.966439 \n",
       "L 235.903125 70.966439 \n",
       "\" clip-path=\"url(#pd5a62a2a66)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5be09a0eaa\" x=\"40.603125\" y=\"70.966439\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(20.878125 74.765658)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 40.603125 32.78791 \n",
       "L 235.903125 32.78791 \n",
       "\" clip-path=\"url(#pd5a62a2a66)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5be09a0eaa\" x=\"40.603125\" y=\"32.78791\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 15 -->\n",
       "      <g transform=\"translate(20.878125 36.587129)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_10\">\n",
       "     <!-- perplexity -->\n",
       "     <g transform=\"translate(14.798437 101.626563)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"229.589844\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"257.373047\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"317.146484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"376.326172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"404.109375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"443.318359\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_17\">\n",
       "    <path d=\"M 40.603125 13.5 \n",
       "L 44.588839 14.213163 \n",
       "L 48.574554 16.168077 \n",
       "L 52.560268 25.77547 \n",
       "L 56.545982 37.057529 \n",
       "L 60.531696 54.834387 \n",
       "L 64.517411 59.633033 \n",
       "L 68.503125 66.508134 \n",
       "L 72.488839 72.232936 \n",
       "L 76.474554 77.128035 \n",
       "L 80.460268 83.581013 \n",
       "L 84.445982 88.297436 \n",
       "L 88.431696 93.829791 \n",
       "L 92.417411 98.495151 \n",
       "L 96.403125 104.768478 \n",
       "L 100.388839 110.040159 \n",
       "L 104.374554 117.237149 \n",
       "L 108.360268 124.792772 \n",
       "L 112.345982 127.77098 \n",
       "L 116.331696 134.229406 \n",
       "L 120.317411 137.402595 \n",
       "L 124.303125 134.668056 \n",
       "L 128.288839 138.73911 \n",
       "L 132.274554 138.870186 \n",
       "L 136.260268 139.081453 \n",
       "L 140.245982 139.081846 \n",
       "L 144.231696 139.255658 \n",
       "L 148.217411 138.936164 \n",
       "L 152.203125 139.285514 \n",
       "L 156.188839 139.419479 \n",
       "L 160.174554 139.355105 \n",
       "L 164.160268 139.393324 \n",
       "L 168.145982 139.412182 \n",
       "L 172.131696 139.431455 \n",
       "L 176.117411 139.428279 \n",
       "L 180.103125 139.387078 \n",
       "L 184.088839 139.430683 \n",
       "L 188.074554 139.5 \n",
       "L 192.060268 139.468257 \n",
       "L 196.045982 139.472716 \n",
       "L 200.031696 139.454257 \n",
       "L 204.017411 139.475074 \n",
       "L 208.003125 139.432604 \n",
       "L 211.988839 139.466875 \n",
       "L 215.974554 139.48762 \n",
       "L 219.960268 139.490837 \n",
       "L 223.945982 139.499248 \n",
       "L 227.931696 139.47738 \n",
       "L 231.917411 139.49972 \n",
       "L 235.903125 139.490411 \n",
       "\" clip-path=\"url(#pd5a62a2a66)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 40.603125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 235.903125 145.8 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 40.603125 7.2 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 173.628125 29.878125 \n",
       "L 228.903125 29.878125 \n",
       "Q 230.903125 29.878125 230.903125 27.878125 \n",
       "L 230.903125 14.2 \n",
       "Q 230.903125 12.2 228.903125 12.2 \n",
       "L 173.628125 12.2 \n",
       "Q 171.628125 12.2 171.628125 14.2 \n",
       "L 171.628125 27.878125 \n",
       "Q 171.628125 29.878125 173.628125 29.878125 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_18\">\n",
       "     <path d=\"M 175.628125 20.298438 \n",
       "L 185.628125 20.298438 \n",
       "L 195.628125 20.298438 \n",
       "\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    </g>\n",
       "    <g id=\"text_11\">\n",
       "     <!-- train -->\n",
       "     <g transform=\"translate(203.628125 23.798438)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pd5a62a2a66\">\n",
       "   <rect x=\"40.603125\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 350x250 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "num_epochs, lr = 500, 2\n",
    "timer = d2l.Timer()\n",
    "timer.start()\n",
    "\n",
    "d2l.train_ch8(model, train_iter, vocab, lr*1.0, num_epochs, device)\n",
    "\n",
    "time = timer.stop()\n",
    "print('LSTM_Timer: %.2f' % time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# 定义GRU层\n",
    "gru_layer = nn.GRU(num_inputs, num_hiddens, num_layers)\n",
    "\n",
    "# 创建RNN模型，使用定义的GRU层和词表大小\n",
    "model_gru = d2l.RNNModel(gru_layer, len(vocab))\n",
    "\n",
    "# 将模型移动到指定的设备（GPU或CPU）\n",
    "model_gru = model_gru.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "perplexity 1.0, 182671.3 tokens/sec on cuda:0\n",
      "time traveller for so it will be convenient to speak of himwas e\n",
      "travelleryou can show black is white by argument said filby\n",
      "GRU_Timer: 31.36\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"252.646875pt\" height=\"183.35625pt\" viewBox=\"0 0 252.646875 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2024-02-14T19:43:19.749980</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 183.35625 \n",
       "L 252.646875 183.35625 \n",
       "L 252.646875 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "L 40.603125 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 76.474554 145.8 \n",
       "L 76.474554 7.2 \n",
       "\" clip-path=\"url(#pb3ef10ce81)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"m76c8d4b2a7\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m76c8d4b2a7\" x=\"76.474554\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(66.930804 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 116.331696 145.8 \n",
       "L 116.331696 7.2 \n",
       "\" clip-path=\"url(#pb3ef10ce81)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m76c8d4b2a7\" x=\"116.331696\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 200 -->\n",
       "      <g transform=\"translate(106.787946 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 156.188839 145.8 \n",
       "L 156.188839 7.2 \n",
       "\" clip-path=\"url(#pb3ef10ce81)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m76c8d4b2a7\" x=\"156.188839\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 300 -->\n",
       "      <g transform=\"translate(146.645089 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 196.045982 145.8 \n",
       "L 196.045982 7.2 \n",
       "\" clip-path=\"url(#pb3ef10ce81)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m76c8d4b2a7\" x=\"196.045982\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 400 -->\n",
       "      <g transform=\"translate(186.502232 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "\" clip-path=\"url(#pb3ef10ce81)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m76c8d4b2a7\" x=\"235.903125\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 500 -->\n",
       "      <g transform=\"translate(226.359375 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(123.025 174.076563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 40.603125 108.897303 \n",
       "L 235.903125 108.897303 \n",
       "\" clip-path=\"url(#pb3ef10ce81)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"m470a99bf3b\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m470a99bf3b\" x=\"40.603125\" y=\"108.897303\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(27.240625 112.696521)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 40.603125 70.481719 \n",
       "L 235.903125 70.481719 \n",
       "\" clip-path=\"url(#pb3ef10ce81)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m470a99bf3b\" x=\"40.603125\" y=\"70.481719\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(20.878125 74.280938)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 40.603125 32.066135 \n",
       "L 235.903125 32.066135 \n",
       "\" clip-path=\"url(#pb3ef10ce81)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m470a99bf3b\" x=\"40.603125\" y=\"32.066135\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 15 -->\n",
       "      <g transform=\"translate(20.878125 35.865354)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_10\">\n",
       "     <!-- perplexity -->\n",
       "     <g transform=\"translate(14.798437 101.626563)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"229.589844\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"257.373047\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"317.146484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"376.326172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"404.109375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"443.318359\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_17\">\n",
       "    <path d=\"M 40.603125 13.5 \n",
       "L 44.588839 56.176568 \n",
       "L 48.574554 67.158668 \n",
       "L 52.560268 71.271421 \n",
       "L 56.545982 81.046797 \n",
       "L 60.531696 87.150913 \n",
       "L 64.517411 94.001742 \n",
       "L 68.503125 101.062799 \n",
       "L 72.488839 109.559677 \n",
       "L 76.474554 117.340861 \n",
       "L 80.460268 124.154406 \n",
       "L 84.445982 132.198807 \n",
       "L 88.431696 136.531518 \n",
       "L 92.417411 138.230655 \n",
       "L 96.403125 138.714397 \n",
       "L 100.388839 138.83984 \n",
       "L 104.374554 138.931089 \n",
       "L 108.360268 139.156721 \n",
       "L 112.345982 139.070769 \n",
       "L 116.331696 139.22693 \n",
       "L 120.317411 139.25411 \n",
       "L 124.303125 139.300969 \n",
       "L 128.288839 139.333936 \n",
       "L 132.274554 139.291275 \n",
       "L 136.260268 139.336282 \n",
       "L 140.245982 139.296521 \n",
       "L 144.231696 139.366795 \n",
       "L 148.217411 139.404563 \n",
       "L 152.203125 139.264904 \n",
       "L 156.188839 139.386787 \n",
       "L 160.174554 139.485782 \n",
       "L 164.160268 139.165775 \n",
       "L 168.145982 139.5 \n",
       "L 172.131696 139.457503 \n",
       "L 176.117411 139.382745 \n",
       "L 180.103125 139.449664 \n",
       "L 184.088839 139.444025 \n",
       "L 188.074554 139.444042 \n",
       "L 192.060268 139.437381 \n",
       "L 196.045982 139.453013 \n",
       "L 200.031696 139.443037 \n",
       "L 204.017411 139.476195 \n",
       "L 208.003125 139.448199 \n",
       "L 211.988839 139.483744 \n",
       "L 215.974554 139.449436 \n",
       "L 219.960268 139.449195 \n",
       "L 223.945982 139.436066 \n",
       "L 227.931696 139.499025 \n",
       "L 231.917411 139.496388 \n",
       "L 235.903125 139.41379 \n",
       "\" clip-path=\"url(#pb3ef10ce81)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 40.603125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 235.903125 145.8 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 40.603125 7.2 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 173.628125 29.878125 \n",
       "L 228.903125 29.878125 \n",
       "Q 230.903125 29.878125 230.903125 27.878125 \n",
       "L 230.903125 14.2 \n",
       "Q 230.903125 12.2 228.903125 12.2 \n",
       "L 173.628125 12.2 \n",
       "Q 171.628125 12.2 171.628125 14.2 \n",
       "L 171.628125 27.878125 \n",
       "Q 171.628125 29.878125 173.628125 29.878125 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_18\">\n",
       "     <path d=\"M 175.628125 20.298438 \n",
       "L 185.628125 20.298438 \n",
       "L 195.628125 20.298438 \n",
       "\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    </g>\n",
       "    <g id=\"text_11\">\n",
       "     <!-- train -->\n",
       "     <g transform=\"translate(203.628125 23.798438)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pb3ef10ce81\">\n",
       "   <rect x=\"40.603125\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 350x250 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "num_epochs, lr = 500, 2\n",
    "timer = d2l.Timer()\n",
    "timer.start()\n",
    "\n",
    "d2l.train_ch8(model_gru, train_iter, vocab, lr*1.0, num_epochs, device)\n",
    "\n",
    "time = timer.stop()\n",
    "print('GRU_Timer: %.2f'%time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;对比双层GRU与双层LSTM的结果可以发现，将LSTM替换为GRU后模型的训练速度有所提高，因为相较于LSTM，GRU只有两个门，具有更少的参数量，因此训练速度更快。精确度方面，由于文本量限制，二者都在该文本上达到了1.0的困惑度，并未出现明显差别。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.3.3\n",
    "\n",
    "如果增加训练数据，能够将困惑度降到多低？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;这里将该作者的另一篇文章[《The War of the Worlds》](https://github.com/datawhalechina/d2l-ai-solutions-manual/blob/master/notebooks/data/the%20war%20of%20the%20worlds.txt)与[《The Time Machine》](https://github.com/datawhalechina/d2l-ai-solutions-manual/blob/master/notebooks/data/timemachine.txt)进行混合，将增加后的文本作为训练数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from d2l import torch as d2l\n",
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_ch8(net, train_iter, vocab, lr, num_epochs, device,\n",
    "              use_random_iter=False):\n",
    "    \"\"\"\n",
    "    训练模型（定义见第8章）\n",
    "    \"\"\"\n",
    "    # 交叉熵损失函数\n",
    "    loss = nn.CrossEntropyLoss()\n",
    "\n",
    "    # 初始化优化器\n",
    "    if isinstance(net, nn.Module):\n",
    "        updater = torch.optim.SGD(net.parameters(), lr)\n",
    "    else:\n",
    "        updater = lambda batch_size: d2l.sgd(net.params, lr, batch_size)\n",
    "\n",
    "    # 预测函数\n",
    "    predict = lambda prefix: d2l.predict_ch8(prefix, 50, net, vocab, device)\n",
    "\n",
    "    # 训练和预测\n",
    "    for epoch in range(num_epochs):\n",
    "        # 调用train_epoch_ch8函数进行一轮训练\n",
    "        ppl, speed = d2l.train_epoch_ch8(net, train_iter, loss, updater, device, use_random_iter)\n",
    "        \n",
    "        # 每10轮打印一次困惑度\n",
    "        if (epoch + 1) % 10 == 0:\n",
    "            print(f'Epoch {epoch + 1}, Perplexity: {ppl:.1f}')\n",
    "\n",
    "    # 打印一个例子的预测结果\n",
    "    print(predict('time traveller'))\n",
    "\n",
    "    # 返回最终的困惑度值\n",
    "    return ppl\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里的 [Mix_data.txt](https://github.com/datawhalechina/d2l-ai-solutions-manual/blob/master/notebooks/data/Mix_data.txt) 是上述两篇文章结合后的文件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def read_Mix():\n",
    "    # 读取文件 Mix_data.txt 中的文本数据\n",
    "    with open('../../notebooks/data/Mix_data.txt', 'r', encoding='utf-8') as f:\n",
    "        lines = f.readlines()\n",
    "    \n",
    "    # 对每一行进行处理，去除非字母字符，转换为小写\n",
    "    return [re.sub('[^A-Za-z]+', ' ', line).strip().lower() for line in lines]\n",
    "\n",
    "def load_corpus_world(max_tokens=-1):\n",
    "    # 通过 read_Mix 函数获取文本数据\n",
    "    lines = read_Mix()\n",
    "    \n",
    "    # 使用 d2l.tokenize 函数将文本行划分为字符\n",
    "    tokens = d2l.tokenize(lines, 'char')\n",
    "    \n",
    "    # 使用 d2l.Vocab 创建词汇表\n",
    "    vocab = d2l.Vocab(tokens)\n",
    "\n",
    "    # 将每个字符转换为词汇表中的索引，形成语料库\n",
    "    corpus = [vocab[token] for line in tokens for token in line]\n",
    "    \n",
    "    # 如果指定了 max_tokens，限制语料库的大小\n",
    "    if max_tokens > 0:\n",
    "        corpus = corpus[:max_tokens]\n",
    "    \n",
    "    # 返回处理后的语料库和词汇表\n",
    "    return corpus, vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "class SeqDataLoader:\n",
    "    def __init__(self, batch_size, num_steps, use_random_iter, max_tokens):\n",
    "        # 根据 use_random_iter 的值选择数据迭代函数\n",
    "        if use_random_iter:\n",
    "            self.data_iter_fn = d2l.seq_data_iter_random\n",
    "        else:\n",
    "            self.data_iter_fn = d2l.seq_data_iter_sequential\n",
    "\n",
    "        # 从文件加载语料库和词汇表\n",
    "        self.corpus, self.vocab = load_corpus_world(max_tokens)\n",
    "        self.batch_size, self.num_steps = batch_size, num_steps\n",
    "\n",
    "    def __iter__(self):\n",
    "        # 返回数据迭代函数的迭代器\n",
    "        return self.data_iter_fn(self.corpus, self.batch_size, self.num_steps)\n",
    "\n",
    "def load_mix_data(batch_size, num_steps, use_random_iter=False, max_tokens=10000):\n",
    "    # 创建 SeqDataLoader 实例作为数据迭代器\n",
    "    data_iter = SeqDataLoader(batch_size, num_steps, use_random_iter, max_tokens)\n",
    "    \n",
    "    # 返回数据迭代器和对应的词汇表\n",
    "    return data_iter, data_iter.vocab\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# 设置训练参数：batch_size 和 num_steps\n",
    "batch_size, num_steps = 32, 35\n",
    "\n",
    "# 调用 load_mix_data 函数加载数据迭代器和词汇表\n",
    "train_iter, vocab = load_mix_data(batch_size, num_steps)\n",
    "\n",
    "# 设置 LSTM 模型的参数\n",
    "num_inputs, num_hiddens, num_layers = len(vocab), 256, 2\n",
    "\n",
    "# 尝试使用 GPU\n",
    "device = d2l.try_gpu()\n",
    "\n",
    "# 创建 LSTM 层\n",
    "lstm_layer = nn.LSTM(num_inputs, num_hiddens, num_layers)\n",
    "\n",
    "# 创建 RNN 模型，将其移动到指定设备\n",
    "model = d2l.RNNModel(lstm_layer, len(vocab))\n",
    "model = model.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "perplexity 1.0, 174917.2 tokens/sec on cuda:0\n",
      "time traveller for so it will be convenient to speak of himwas e\n",
      "travelleryou can show black is white by argument said filby\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"252.646875pt\" height=\"183.35625pt\" viewBox=\"0 0 252.646875 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2024-02-14T19:43:51.872906</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 183.35625 \n",
       "L 252.646875 183.35625 \n",
       "L 252.646875 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "L 40.603125 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 76.474554 145.8 \n",
       "L 76.474554 7.2 \n",
       "\" clip-path=\"url(#p55c83f980c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"m78879f76fb\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m78879f76fb\" x=\"76.474554\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(66.930804 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 116.331696 145.8 \n",
       "L 116.331696 7.2 \n",
       "\" clip-path=\"url(#p55c83f980c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m78879f76fb\" x=\"116.331696\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 200 -->\n",
       "      <g transform=\"translate(106.787946 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 156.188839 145.8 \n",
       "L 156.188839 7.2 \n",
       "\" clip-path=\"url(#p55c83f980c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m78879f76fb\" x=\"156.188839\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 300 -->\n",
       "      <g transform=\"translate(146.645089 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 196.045982 145.8 \n",
       "L 196.045982 7.2 \n",
       "\" clip-path=\"url(#p55c83f980c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m78879f76fb\" x=\"196.045982\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 400 -->\n",
       "      <g transform=\"translate(186.502232 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "\" clip-path=\"url(#p55c83f980c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m78879f76fb\" x=\"235.903125\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 500 -->\n",
       "      <g transform=\"translate(226.359375 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(123.025 174.076563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 40.603125 109.106449 \n",
       "L 235.903125 109.106449 \n",
       "\" clip-path=\"url(#p55c83f980c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"m2d3af15d6a\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m2d3af15d6a\" x=\"40.603125\" y=\"109.106449\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(27.240625 112.905667)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 40.603125 70.917326 \n",
       "L 235.903125 70.917326 \n",
       "\" clip-path=\"url(#p55c83f980c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m2d3af15d6a\" x=\"40.603125\" y=\"70.917326\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(20.878125 74.716545)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 40.603125 32.728204 \n",
       "L 235.903125 32.728204 \n",
       "\" clip-path=\"url(#p55c83f980c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m2d3af15d6a\" x=\"40.603125\" y=\"32.728204\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 15 -->\n",
       "      <g transform=\"translate(20.878125 36.527423)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_10\">\n",
       "     <!-- perplexity -->\n",
       "     <g transform=\"translate(14.798437 101.626563)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"229.589844\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"257.373047\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"317.146484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"376.326172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"404.109375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"443.318359\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_17\">\n",
       "    <path d=\"M 40.603125 13.5 \n",
       "L 44.588839 13.891725 \n",
       "L 48.574554 17.043347 \n",
       "L 52.560268 26.997687 \n",
       "L 56.545982 37.606395 \n",
       "L 60.531696 57.98678 \n",
       "L 64.517411 61.557709 \n",
       "L 68.503125 67.922506 \n",
       "L 72.488839 73.65011 \n",
       "L 76.474554 77.668523 \n",
       "L 80.460268 82.823138 \n",
       "L 84.445982 88.920825 \n",
       "L 88.431696 94.157575 \n",
       "L 92.417411 100.031172 \n",
       "L 96.403125 104.747488 \n",
       "L 100.388839 111.100955 \n",
       "L 104.374554 117.005761 \n",
       "L 108.360268 124.318523 \n",
       "L 112.345982 129.895146 \n",
       "L 116.331696 133.366201 \n",
       "L 120.317411 136.884818 \n",
       "L 124.303125 138.568973 \n",
       "L 128.288839 138.677188 \n",
       "L 132.274554 138.806209 \n",
       "L 136.260268 139.020607 \n",
       "L 140.245982 139.243238 \n",
       "L 144.231696 139.337078 \n",
       "L 148.217411 139.297841 \n",
       "L 152.203125 139.296321 \n",
       "L 156.188839 139.311928 \n",
       "L 160.174554 139.367792 \n",
       "L 164.160268 139.332699 \n",
       "L 168.145982 139.384324 \n",
       "L 172.131696 139.372046 \n",
       "L 176.117411 139.399289 \n",
       "L 180.103125 139.465715 \n",
       "L 184.088839 139.456937 \n",
       "L 188.074554 139.420087 \n",
       "L 192.060268 139.453872 \n",
       "L 196.045982 139.424669 \n",
       "L 200.031696 139.5 \n",
       "L 204.017411 139.490525 \n",
       "L 208.003125 139.45109 \n",
       "L 211.988839 139.308924 \n",
       "L 215.974554 139.384421 \n",
       "L 219.960268 139.409354 \n",
       "L 223.945982 139.436748 \n",
       "L 227.931696 139.423729 \n",
       "L 231.917411 139.449838 \n",
       "L 235.903125 139.454213 \n",
       "\" clip-path=\"url(#p55c83f980c)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 40.603125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 235.903125 145.8 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 40.603125 145.8 \n",
       "L 235.903125 145.8 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 40.603125 7.2 \n",
       "L 235.903125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 173.628125 29.878125 \n",
       "L 228.903125 29.878125 \n",
       "Q 230.903125 29.878125 230.903125 27.878125 \n",
       "L 230.903125 14.2 \n",
       "Q 230.903125 12.2 228.903125 12.2 \n",
       "L 173.628125 12.2 \n",
       "Q 171.628125 12.2 171.628125 14.2 \n",
       "L 171.628125 27.878125 \n",
       "Q 171.628125 29.878125 173.628125 29.878125 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_18\">\n",
       "     <path d=\"M 175.628125 20.298438 \n",
       "L 185.628125 20.298438 \n",
       "L 195.628125 20.298438 \n",
       "\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "    </g>\n",
       "    <g id=\"text_11\">\n",
       "     <!-- train -->\n",
       "     <g transform=\"translate(203.628125 23.798438)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p55c83f980c\">\n",
       "   <rect x=\"40.603125\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 350x250 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 设置训练参数：num_epochs（训练轮数）和 lr（学习率）\n",
    "num_epochs, lr = 500, 2\n",
    "\n",
    "# 调用 d2l.train_ch8 函数进行模型训练\n",
    "d2l.train_ch8(model, train_iter, vocab, lr*1.0, num_epochs, device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;当数据量为96135字时，LSTM模型的困惑度仍然能够降低至1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.3.4\n",
    "\n",
    "在为文本建模时，是否可以将不同作者的源数据合并？有何优劣呢？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;在为文本建模时，将不同作者的源数据合并是可以的，但需要考虑一些因素。  \n",
    "- 优点：\n",
    "    - 增加训练数据量：将不同作者的源数据合并，可以增加训练数据量，从而更好地训练模型。[1,3]\n",
    "    - 提高模型泛化能力：不同作者的风格、语言等可能不同，将这些数据合并可以让模型更好地学习到不同的文本特征，提高模型的泛化能力。  \n",
    "\n",
    "- 缺点：\n",
    "    - 可能影响模型训练效果：合并不同作者的数据，可能会引入更多的噪音，使得模型更难学习到有用的信息，从而影响训练效果。[2]  \n",
    "    - 可能会引入偏见：如果不同作者的数据不平衡，比如某一位作者的数据量比其他作者多很多，那么训练出来的模型可能会对该作者的文本表现更好，从而引入偏见。\n",
    "\n",
    "&emsp;&emsp;综上所述，将不同作者的源数据合并在一起是可以的，但需要在合并前考虑好数据的平衡性和质量，并对数据进行适当的预处理，以确保训练出来的模型具有较好的泛化能力和稳定性。\n",
    "\n",
    "---\n",
    "\n",
    "[1] Peng, N., & Dredze, M. (2016). Improving named entity recognition for chinese social media with word segmentation representation learning. arXiv preprint arXiv:1603.00786.\\\n",
    "[2] Kadadi, A., Agrawal, R., Nyamful, C., & Atiq, R. (2014, October). Challenges of data integration and interoperability in big data. In 2014 IEEE international conference on big data (big data) (pp. 38-40). IEEE.\\\n",
    "[3] Kobayashi, H., Sakamoto, M., Nakashima, A., Hirayu, K., & Ikawa, S. (2023). U.S. Patent Application No. 17/455,514."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 9.4 双向循环神经网络 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.4.1\n",
    "\n",
    "如果不同方向使用不同数量的隐藏单位，$\\mathbf{H_t}$ 的形状会发生怎样的变化？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;在双向循环神经网络中，最终隐藏状态 $H_t$ 的维度由前向循环神经网络和反向循环神经网络中的隐藏层进行 $Concat$ 得来，而通常前向和反向循环神经网络均选用相同的隐藏层数量，即    $\\overrightarrow{H_t}=\\overleftarrow{H_t}$ ，因此直接进行翻倍，即可获得最终形状 $H_t=2 * \\overrightarrow{H_t} = 2 * \\overleftarrow{H_t}$ 。  \n",
    "&emsp;&emsp;而当不同方向使用不同数量的隐藏单位时，此时前向和反向的隐藏单元数量不同， $H_t$ 的维度并不等于前向或反向循环神经网络中隐藏层数量的 $2$ 倍，此时需要将前向和反向的隐藏层维度拼接相加，获得最终 $H_t$ 的维度，即 $H_t = \\overrightarrow{H_t} + \\overleftarrow{H_t}$ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class BiLSTM(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size_forward, hidden_size_backward, num_layers=1):\n",
    "        super(BiLSTM, self).__init__()\n",
    "\n",
    "        # 定义前向LSTM层\n",
    "        # batch_first=True 意味着输入形状为 [batch_size, seq_length, feature_dim]\n",
    "        self.lstm_forward = nn.LSTM(input_size, hidden_size_forward, num_layers=num_layers, batch_first=True)\n",
    "        \n",
    "        # 定义反向LSTM层\n",
    "        self.lstm_backward = nn.LSTM(input_size, hidden_size_backward, num_layers=num_layers, batch_first=True)\n",
    "\n",
    "    def forward(self, x, x_lengths):\n",
    "        # 前向部分\n",
    "        # pack_padded_sequence() 负责将可变长度序列进行打包\n",
    "        packed_x_forward = nn.utils.rnn.pack_padded_sequence(x, x_lengths, batch_first=True, enforce_sorted=False)\n",
    "        packed_h_forward, _ = self.lstm_forward(packed_x_forward)\n",
    "        \n",
    "        # pad_packed_sequence() 负责将填充的序列解包，以便在后续计算时去掉无用的填充部分\n",
    "        h_forward, _ = nn.utils.rnn.pad_packed_sequence(packed_h_forward, batch_first=True)\n",
    "\n",
    "        # 反向部分\n",
    "        # flip() 用于颠倒序列中的元素，以便进行反向LSTM计算\n",
    "        packed_x_backward = nn.utils.rnn.pack_padded_sequence(torch.flip(x, [1]), x_lengths, batch_first=True, enforce_sorted=False)\n",
    "        packed_h_backward, _ = self.lstm_backward(packed_x_backward)\n",
    "        \n",
    "        # pad_packed_sequence() 解包填充的序列\n",
    "        h_backward, _ = nn.utils.rnn.pad_packed_sequence(packed_h_backward, batch_first=True)\n",
    "        \n",
    "        # 反转反向LSTM的输出以匹配正向LSTM\n",
    "        h_backward = torch.flip(h_backward, [1])\n",
    "\n",
    "        # Concatenate 正向和反向LSTM的输出\n",
    "        h = torch.cat((h_forward, h_backward), dim=2)\n",
    "\n",
    "        return h"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BiLSTM(\n",
      "  (lstm_forward): LSTM(2, 10, batch_first=True)\n",
      "  (lstm_backward): LSTM(2, 8, batch_first=True)\n",
      ")\n",
      "torch.Size([3, 5, 18])\n"
     ]
    }
   ],
   "source": [
    "# 定义模型和输入参数\n",
    "batch_size = 3\n",
    "seq_length = 5\n",
    "input_size = 2\n",
    "hidden_size_forward = 10\n",
    "hidden_size_backward = 8\n",
    "\n",
    "# 构造不同长度的输入序列\n",
    "x_lengths = [3, 5, 2]\n",
    "x = [torch.randn(x_lengths[i], input_size) for i in range(batch_size)]\n",
    "\n",
    "# 使用 pad_sequence 进行填充，使得输入序列具有相同的长度\n",
    "x = nn.utils.rnn.pad_sequence(x, batch_first=True)\n",
    "\n",
    "# 创建 BiLSTM 模型\n",
    "model = BiLSTM(input_size, hidden_size_forward, hidden_size_backward)\n",
    "print(model)\n",
    "\n",
    "# 进行前向传播\n",
    "h = model(x, x_lengths)\n",
    "print(h.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;从这里我们可以发现到当前向和反向的隐藏节点不同的时候， $H_t$ 最终的形状会是 $\\overrightarrow{H_t} + \\overleftarrow{H_t}$。即前向中的 $10$ 和反向中的 $8$ 得出 $10+8=18$。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.4.2\n",
    "\n",
    "设计一个具有多个隐藏层的双向循环神经网络。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RNNModel(\n",
      "  (rnn): LSTM(28, 256, num_layers=4, bidirectional=True)\n",
      "  (linear): Linear(in_features=512, out_features=28, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from d2l import torch as d2l\n",
    "\n",
    "# 加载数据\n",
    "batch_size, num_steps, device = 32, 35, d2l.try_gpu()\n",
    "train_iter, vocab = d2l.load_data_time_machine(batch_size, num_steps)\n",
    "\n",
    "# 定义模型参数\n",
    "vocab_size, num_hiddens, num_layers = len(vocab), 256, 4\n",
    "num_inputs = vocab_size\n",
    "\n",
    "# 创建双向LSTM层\n",
    "lstm_layer = nn.LSTM(num_inputs, num_hiddens, num_layers, bidirectional=True)\n",
    "\n",
    "# 创建 RNN 模型，将其移动到指定设备\n",
    "model = d2l.RNNModel(lstm_layer, len(vocab))\n",
    "model = model.to(device)\n",
    "\n",
    "# 打印模型结构\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;在上述模型中，隐藏层的数量被设置为4，实现了一个四层的双向循环神经网络。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.4.3\n",
    "\n",
    "在自然语言中一词多义很常见。例如，“bank” 一词在不同的上下文 “I went to the bank to deposit cash” 和 “I went to the bank to sit down” 中有不同的含义。如何设计一个神经网络模型，使其在给定上下文序列和单词的情况下，返回该单词在此上下文中的向量表示？哪种类型的神经网络架构更适合处理一词多义？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;处理一词多义的常见方法是结合上下文信息来对该位置的单词意思进行确认，例如在神经网络中，可以采用双向循环神经网络模型或者Transformer，将上下文中的单词序列映射到单词的向量表示。这些模型可以学到上下文中单词的复杂交互，以此实现根据上下文内容来动态生成单词的对应向量表示。  \n",
    "\n",
    "&emsp;&emsp;这些上下文感知的双向模型能够更好地捕捉单词在不同上下文中的含义。例如，在双向LSTM中，每个单词的向量表示取决于整个上下文序列，因此可以更好地捕捉单词在上下文中的含义。\n",
    "\n",
    "---\n",
    "\n",
    "[1] Sarzynska-Wawer, J., Wawer, A., Pawlak, A., Szymanowska, J., Stefaniak, I., Jarkiewicz, M., & Okruszek, L. (2021). Detecting formal thought disorder by deep contextualized word representations. Psychiatry Research, 304, 114135.\\\n",
    "[2] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 9.5 机器翻译与数据集 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.5.1\n",
    "\n",
    "在`load_data_nmt`函数中尝试不同的`num_examples`参数值。这对源语言和目标语言的词表大小有何影响？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "#@save\n",
    "d2l.DATA_HUB['fra-eng'] = (d2l.DATA_URL + 'fra-eng.zip',\n",
    "                           '94646ad1522d915e7b0f9296181140edcf86a4f5')\n",
    "\n",
    "#@save\n",
    "def read_data_nmt():\n",
    "    \"\"\"载入“英语－法语”数据集\"\"\"\n",
    "    data_dir = d2l.download_extract('fra-eng')\n",
    "    with open(os.path.join(data_dir, 'fra.txt'), 'r',\n",
    "             encoding='utf-8') as f:\n",
    "        return f.read()\n",
    "\n",
    "#@save\n",
    "def load_data_nmt(batch_size, num_steps, num_examples):\n",
    "    \"\"\"返回翻译数据集的迭代器和词表\"\"\"\n",
    "    text = d2l.preprocess_nmt(read_data_nmt())\n",
    "    source, target = d2l.tokenize_nmt(text, num_examples)\n",
    "    src_vocab = d2l.Vocab(source, min_freq=2,\n",
    "                          reserved_tokens=['<pad>', '<bos>', '<eos>'])\n",
    "    tgt_vocab = d2l.Vocab(target, min_freq=2,\n",
    "                          reserved_tokens=['<pad>', '<bos>', '<eos>'])\n",
    "    src_array, src_valid_len = d2l.build_array_nmt(source, src_vocab, num_steps)\n",
    "    tgt_array, tgt_valid_len = d2l.build_array_nmt(target, tgt_vocab, num_steps)\n",
    "    data_arrays = (src_array, src_valid_len, tgt_array, tgt_valid_len)\n",
    "    data_iter = d2l.load_array(data_arrays, batch_size)\n",
    "    return data_iter, src_vocab, tgt_vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[40, 69, 102, 130, 159, 184] [40, 67, 107, 125, 163, 201]\n"
     ]
    }
   ],
   "source": [
    "num_examples = [100, 200, 300, 400, 500, 600]\n",
    "src = []\n",
    "tgt = []\n",
    "for i in num_examples:\n",
    "    train_iter_2, src_vocab, tgt_vocab = load_data_nmt(batch_size=2, num_steps=8, num_examples=i)\n",
    "    src.append(len(src_vocab))\n",
    "    tgt.append(len(tgt_vocab))\n",
    "\n",
    "print(src, tgt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"388.965625pt\" height=\"211.354375pt\" viewBox=\"0 0 388.965625 211.354375\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2023-05-09T15:28:25.990597</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 211.354375 \n",
       "L 388.965625 211.354375 \n",
       "L 388.965625 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 46.965625 173.52 \n",
       "L 381.765625 173.52 \n",
       "L 381.765625 7.2 \n",
       "L 46.965625 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 66.902623 173.52 \n",
       "L 71.621439 173.52 \n",
       "L 71.621439 141.997612 \n",
       "L 66.902623 141.997612 \n",
       "z\n",
       "\" clip-path=\"url(#p470aef8832)\" style=\"fill: #1f77b4\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 125.887824 173.52 \n",
       "L 130.60664 173.52 \n",
       "L 130.60664 119.143881 \n",
       "L 125.887824 119.143881 \n",
       "z\n",
       "\" clip-path=\"url(#p470aef8832)\" style=\"fill: #1f77b4\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 184.873025 173.52 \n",
       "L 189.591841 173.52 \n",
       "L 189.591841 93.13791 \n",
       "L 184.873025 93.13791 \n",
       "z\n",
       "\" clip-path=\"url(#p470aef8832)\" style=\"fill: #1f77b4\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 243.858225 173.52 \n",
       "L 248.577041 173.52 \n",
       "L 248.577041 71.072239 \n",
       "L 243.858225 71.072239 \n",
       "z\n",
       "\" clip-path=\"url(#p470aef8832)\" style=\"fill: #1f77b4\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_7\">\n",
       "    <path d=\"M 302.843426 173.52 \n",
       "L 307.562242 173.52 \n",
       "L 307.562242 48.218507 \n",
       "L 302.843426 48.218507 \n",
       "z\n",
       "\" clip-path=\"url(#p470aef8832)\" style=\"fill: #1f77b4\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_8\">\n",
       "    <path d=\"M 361.828627 173.52 \n",
       "L 366.547443 173.52 \n",
       "L 366.547443 28.517015 \n",
       "L 361.828627 28.517015 \n",
       "z\n",
       "\" clip-path=\"url(#p470aef8832)\" style=\"fill: #1f77b4\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_9\">\n",
       "    <path d=\"M 66.902623 173.52 \n",
       "L 62.183807 173.52 \n",
       "L 62.183807 141.997612 \n",
       "L 66.902623 141.997612 \n",
       "z\n",
       "\" clip-path=\"url(#p470aef8832)\" style=\"fill: #ff7f0e\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_10\">\n",
       "    <path d=\"M 125.887824 173.52 \n",
       "L 121.169008 173.52 \n",
       "L 121.169008 120.72 \n",
       "L 125.887824 120.72 \n",
       "z\n",
       "\" clip-path=\"url(#p470aef8832)\" style=\"fill: #ff7f0e\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_11\">\n",
       "    <path d=\"M 184.873025 173.52 \n",
       "L 180.154209 173.52 \n",
       "L 180.154209 89.197612 \n",
       "L 184.873025 89.197612 \n",
       "z\n",
       "\" clip-path=\"url(#p470aef8832)\" style=\"fill: #ff7f0e\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_12\">\n",
       "    <path d=\"M 243.858225 173.52 \n",
       "L 239.139409 173.52 \n",
       "L 239.139409 75.012537 \n",
       "L 243.858225 75.012537 \n",
       "z\n",
       "\" clip-path=\"url(#p470aef8832)\" style=\"fill: #ff7f0e\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_13\">\n",
       "    <path d=\"M 302.843426 173.52 \n",
       "L 298.12461 173.52 \n",
       "L 298.12461 45.066269 \n",
       "L 302.843426 45.066269 \n",
       "z\n",
       "\" clip-path=\"url(#p470aef8832)\" style=\"fill: #ff7f0e\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_14\">\n",
       "    <path d=\"M 361.828627 173.52 \n",
       "L 357.109811 173.52 \n",
       "L 357.109811 15.12 \n",
       "L 361.828627 15.12 \n",
       "z\n",
       "\" clip-path=\"url(#p470aef8832)\" style=\"fill: #ff7f0e\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path id=\"m155ffabb38\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m155ffabb38\" x=\"66.902623\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(57.358873 188.118437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m155ffabb38\" x=\"125.887824\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 200 -->\n",
       "      <g transform=\"translate(116.344074 188.118437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m155ffabb38\" x=\"184.873025\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 300 -->\n",
       "      <g transform=\"translate(175.329275 188.118437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m155ffabb38\" x=\"243.858225\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 400 -->\n",
       "      <g transform=\"translate(234.314475 188.118437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m155ffabb38\" x=\"302.843426\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 500 -->\n",
       "      <g transform=\"translate(293.299676 188.118437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m155ffabb38\" x=\"361.828627\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- 600 -->\n",
       "      <g transform=\"translate(352.284877 188.118437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n",
       "Q 1688 2584 1439 2293 \n",
       "Q 1191 2003 1191 1497 \n",
       "Q 1191 994 1439 701 \n",
       "Q 1688 409 2113 409 \n",
       "Q 2538 409 2786 701 \n",
       "Q 3034 994 3034 1497 \n",
       "Q 3034 2003 2786 2293 \n",
       "Q 2538 2584 2113 2584 \n",
       "z\n",
       "M 3366 4563 \n",
       "L 3366 3988 \n",
       "Q 3128 4100 2886 4159 \n",
       "Q 2644 4219 2406 4219 \n",
       "Q 1781 4219 1451 3797 \n",
       "Q 1122 3375 1075 2522 \n",
       "Q 1259 2794 1537 2939 \n",
       "Q 1816 3084 2150 3084 \n",
       "Q 2853 3084 3261 2657 \n",
       "Q 3669 2231 3669 1497 \n",
       "Q 3669 778 3244 343 \n",
       "Q 2819 -91 2113 -91 \n",
       "Q 1303 -91 875 529 \n",
       "Q 447 1150 447 2328 \n",
       "Q 447 3434 972 4092 \n",
       "Q 1497 4750 2381 4750 \n",
       "Q 2619 4750 2861 4703 \n",
       "Q 3103 4656 3366 4563 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_7\">\n",
       "     <!-- num_examples -->\n",
       "     <g transform=\"translate(176.530469 201.796562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-75\" d=\"M 544 1381 \n",
       "L 544 3500 \n",
       "L 1119 3500 \n",
       "L 1119 1403 \n",
       "Q 1119 906 1312 657 \n",
       "Q 1506 409 1894 409 \n",
       "Q 2359 409 2629 706 \n",
       "Q 2900 1003 2900 1516 \n",
       "L 2900 3500 \n",
       "L 3475 3500 \n",
       "L 3475 0 \n",
       "L 2900 0 \n",
       "L 2900 538 \n",
       "Q 2691 219 2414 64 \n",
       "Q 2138 -91 1772 -91 \n",
       "Q 1169 -91 856 284 \n",
       "Q 544 659 544 1381 \n",
       "z\n",
       "M 1991 3584 \n",
       "L 1991 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n",
       "Q 3544 3216 3844 3400 \n",
       "Q 4144 3584 4550 3584 \n",
       "Q 5097 3584 5394 3201 \n",
       "Q 5691 2819 5691 2113 \n",
       "L 5691 0 \n",
       "L 5113 0 \n",
       "L 5113 2094 \n",
       "Q 5113 2597 4934 2840 \n",
       "Q 4756 3084 4391 3084 \n",
       "Q 3944 3084 3684 2787 \n",
       "Q 3425 2491 3425 1978 \n",
       "L 3425 0 \n",
       "L 2847 0 \n",
       "L 2847 2094 \n",
       "Q 2847 2600 2669 2842 \n",
       "Q 2491 3084 2119 3084 \n",
       "Q 1678 3084 1418 2786 \n",
       "Q 1159 2488 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1356 3278 1631 3431 \n",
       "Q 1906 3584 2284 3584 \n",
       "Q 2666 3584 2933 3390 \n",
       "Q 3200 3197 3328 2828 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-5f\" d=\"M 3263 -1063 \n",
       "L 3263 -1509 \n",
       "L -63 -1509 \n",
       "L -63 -1063 \n",
       "L 3263 -1063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
       "L 2247 1797 \n",
       "L 3578 0 \n",
       "L 2900 0 \n",
       "L 1881 1375 \n",
       "L 863 0 \n",
       "L 184 0 \n",
       "L 1544 1831 \n",
       "L 300 3500 \n",
       "L 978 3500 \n",
       "L 1906 2253 \n",
       "L 2834 3500 \n",
       "L 3513 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-75\" x=\"63.378906\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"126.757812\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-5f\" x=\"224.169922\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"274.169922\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-78\" x=\"333.943359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"393.123047\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"454.402344\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"551.814453\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"615.291016\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"643.074219\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"704.597656\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <defs>\n",
       "       <path id=\"mc69851257f\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc69851257f\" x=\"46.965625\" y=\"173.52\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(33.603125 177.319219)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc69851257f\" x=\"46.965625\" y=\"134.117015\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 50 -->\n",
       "      <g transform=\"translate(27.240625 137.916234)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc69851257f\" x=\"46.965625\" y=\"94.71403\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(20.878125 98.513249)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc69851257f\" x=\"46.965625\" y=\"55.311045\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 150 -->\n",
       "      <g transform=\"translate(20.878125 59.110264)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc69851257f\" x=\"46.965625\" y=\"15.90806\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 200 -->\n",
       "      <g transform=\"translate(20.878125 19.707278)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- Count -->\n",
       "     <g transform=\"translate(14.798438 105.208437)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-43\" d=\"M 4122 4306 \n",
       "L 4122 3641 \n",
       "Q 3803 3938 3442 4084 \n",
       "Q 3081 4231 2675 4231 \n",
       "Q 1875 4231 1450 3742 \n",
       "Q 1025 3253 1025 2328 \n",
       "Q 1025 1406 1450 917 \n",
       "Q 1875 428 2675 428 \n",
       "Q 3081 428 3442 575 \n",
       "Q 3803 722 4122 1019 \n",
       "L 4122 359 \n",
       "Q 3791 134 3420 21 \n",
       "Q 3050 -91 2638 -91 \n",
       "Q 1578 -91 968 557 \n",
       "Q 359 1206 359 2328 \n",
       "Q 359 3453 968 4101 \n",
       "Q 1578 4750 2638 4750 \n",
       "Q 3056 4750 3426 4639 \n",
       "Q 3797 4528 4122 4306 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-43\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"69.824219\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-75\" x=\"131.005859\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6e\" x=\"194.384766\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"257.763672\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_15\">\n",
       "    <path d=\"M 46.965625 173.52 \n",
       "L 46.965625 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_16\">\n",
       "    <path d=\"M 381.765625 173.52 \n",
       "L 381.765625 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_17\">\n",
       "    <path d=\"M 46.965625 173.52 \n",
       "L 381.765625 173.52 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_18\">\n",
       "    <path d=\"M 46.965625 7.2 \n",
       "L 381.765625 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_19\">\n",
       "     <path d=\"M 53.965625 44.55625 \n",
       "L 119.16875 44.55625 \n",
       "Q 121.16875 44.55625 121.16875 42.55625 \n",
       "L 121.16875 14.2 \n",
       "Q 121.16875 12.2 119.16875 12.2 \n",
       "L 53.965625 12.2 \n",
       "Q 51.965625 12.2 51.965625 14.2 \n",
       "L 51.965625 42.55625 \n",
       "Q 51.965625 44.55625 53.965625 44.55625 \n",
       "z\n",
       "\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n",
       "    </g>\n",
       "    <g id=\"patch_20\">\n",
       "     <path d=\"M 55.965625 23.798437 \n",
       "L 75.965625 23.798437 \n",
       "L 75.965625 16.798437 \n",
       "L 55.965625 16.798437 \n",
       "z\n",
       "\" style=\"fill: #1f77b4\"/>\n",
       "    </g>\n",
       "    <g id=\"text_14\">\n",
       "     <!-- source -->\n",
       "     <g transform=\"translate(83.965625 23.798437)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-73\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"52.099609\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-75\" x=\"113.28125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"176.660156\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"215.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"270.503906\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"patch_21\">\n",
       "     <path d=\"M 55.965625 38.476562 \n",
       "L 75.965625 38.476562 \n",
       "L 75.965625 31.476562 \n",
       "L 55.965625 31.476562 \n",
       "z\n",
       "\" style=\"fill: #ff7f0e\"/>\n",
       "    </g>\n",
       "    <g id=\"text_15\">\n",
       "     <!-- target -->\n",
       "     <g transform=\"translate(83.965625 38.476562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \n",
       "Q 2906 2416 2648 2759 \n",
       "Q 2391 3103 1925 3103 \n",
       "Q 1463 3103 1205 2759 \n",
       "Q 947 2416 947 1791 \n",
       "Q 947 1169 1205 825 \n",
       "Q 1463 481 1925 481 \n",
       "Q 2391 481 2648 825 \n",
       "Q 2906 1169 2906 1791 \n",
       "z\n",
       "M 3481 434 \n",
       "Q 3481 -459 3084 -895 \n",
       "Q 2688 -1331 1869 -1331 \n",
       "Q 1566 -1331 1297 -1286 \n",
       "Q 1028 -1241 775 -1147 \n",
       "L 775 -588 \n",
       "Q 1028 -725 1275 -790 \n",
       "Q 1522 -856 1778 -856 \n",
       "Q 2344 -856 2625 -561 \n",
       "Q 2906 -266 2906 331 \n",
       "L 2906 616 \n",
       "Q 2728 306 2450 153 \n",
       "Q 2172 0 1784 0 \n",
       "Q 1141 0 747 490 \n",
       "Q 353 981 353 1791 \n",
       "Q 353 2603 747 3093 \n",
       "Q 1141 3584 1784 3584 \n",
       "Q 2172 3584 2450 3431 \n",
       "Q 2728 3278 2906 2969 \n",
       "L 2906 3500 \n",
       "L 3481 3500 \n",
       "L 3481 434 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"39.208984\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"100.488281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-67\" x=\"139.851562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"203.328125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-74\" x=\"264.851562\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p470aef8832\">\n",
       "   <rect x=\"46.965625\" y=\"7.2\" width=\"334.8\" height=\"166.32\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 600x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "ax.bar(num_examples, src, width=8, align='edge', label='source')\n",
    "ax.bar(num_examples, tgt, width=-8, align='edge', label='target')\n",
    "ax.set_xlabel('num_examples')\n",
    "ax.set_ylabel('Count')\n",
    "ax.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;在tokenize_nmt函数中，num_examples主要用于规定进行词元化的序列的长度，因此num_examples越大，所获得的词表越大。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.5.2\n",
    "\n",
    "某些语言（例如中文和日语）的文本没有单词边界指示符（例如空格）。对于这种情况，单词级词元化仍然是个好主意吗？为什么？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;以中文为例，中文中不存在类似空格这样的单词边界指示符，但是中文可采用基于词典、基于规则或者基于统计的分词方法，针对中文文本进行分词，以此实现中文的词元化处理。  \n",
    "&emsp;&emsp;这种方法可以将中文文本划分为单独的词语，以实现对中文的精确处理。例如在词义消歧时，单词级的中文分词可以帮助我们更准确的确定词语的上下文，以此来确定歧义词的准确意义。并且在信息检索任务中，很多检索都是基于关键词展开的，对中文进行单词级词元化，可以便于在索引中进行文本匹配。因此，即便中文和日语这些语言中没有单词边界指示符，但在进行自然语言处理时，单词级的词元化仍是非常关键的一步。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 9.6 编码器-解码器架构"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.6.1\n",
    "\n",
    "假设我们使用神经网络来实现“编码器－解码器”架构，那么编码器和解码器必须是同一类型的神经网络吗？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;不一定。虽然编码器和解码器通常是相同类型的神经网络，但它们可以是不同的网络架构。事实上，有些最先进的神经网络模型在编码器和解码器之间使用了不同的架构。例如，使用CNN作为编码器，然后使用RNN或Transformer作为解码器的架构在机器翻译任务中很常见。  \n",
    "&emsp;&emsp;另外，编码器和解码器也可以在不同的领域应用不同类型的神经网络。例如，在图像到文本任务中，可以使用CNN作为编码器，将图像编码为特征向量，然后使用RNN或Transformer作为解码器生成相应的自然语言描述。因此，编码器和解码器可以是同一类型的神经网络，也可以是不同类型的神经网络，它实质上只是一种抽象的结构，该结构可以适用于多种神经网络，具体选择需要取决于具体的任务需求和设计思路。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.6.2\n",
    "\n",
    "除了机器翻译，还有其它可以适用于”编码器－解码器“架构的应用吗？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;目前，“编码器-解码器”结构已经在多种任务中取得了优秀的研究成果，并非仅仅局限于机器翻译一类任务中。例如语音识别任务，可以采用编码器接受音频信号，并将其编码为特征向量，再由解码器将其转化为文本内容；在图像生成任务中，编码器可以将输入的图像或文本内容进行编码，再由解码器根据编码的特征向量生成对应的图像；此外还有文本生成任务，编码器将输入文本进行编码后，再由解码器使用这些编码后的特征向量进行解码，生成与原始文本内容相似的文本。  \n",
    "&emsp;&emsp;实际上，“编码器-解码器”是一个通用的抽象架构，其基本思想是将输入数据编码为一个便于神经网络学习和理解的中间表示，再由解码器将中间表示转换为所需要的输出。这种架构可以根据具体需求适用于不同的实际场景中。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 9.7 序列到序列的学习（seq2seq） "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.7.1\n",
    "\n",
    "试着通过调整超参数来改善翻译效果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;尝试将隐藏层和Embedding层维度修改为256，同时将batch_size修改为256，学习率降低至0.001"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import collections\n",
    "import math\n",
    "import torch\n",
    "from torch import nn\n",
    "from d2l import torch as d2l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "class Seq2SeqDecoder(d2l.Decoder):\n",
    "    \"\"\"用于序列到序列学习的循环神经网络解码器\"\"\"\n",
    "    def __init__(self, vocab_size, embed_size, num_hiddens, num_layers,\n",
    "                 dropout=0, **kwargs):\n",
    "        super(Seq2SeqDecoder, self).__init__(**kwargs)\n",
    "        self.embedding = nn.Embedding(vocab_size, embed_size)\n",
    "        self.rnn = nn.GRU(embed_size + num_hiddens, num_hiddens, num_layers,\n",
    "                          dropout=dropout)\n",
    "        self.dense = nn.Linear(num_hiddens, vocab_size)\n",
    "\n",
    "    def init_state(self, enc_outputs, *args):\n",
    "        return enc_outputs[1]\n",
    "\n",
    "    def forward(self, X, state):\n",
    "        # 输出'X'的形状：(batch_size,num_steps,embed_size)\n",
    "        X = self.embedding(X).permute(1, 0, 2)\n",
    "        # 广播context，使其具有与X相同的num_steps\n",
    "        context = state[-1].repeat(X.shape[0], 1, 1)\n",
    "        X_and_context = torch.cat((X, context), 2)\n",
    "        output, state = self.rnn(X_and_context, state)\n",
    "        output = self.dense(output).permute(1, 0, 2)\n",
    "        # output的形状:(batch_size,num_steps,vocab_size)\n",
    "        # state的形状:(num_layers,batch_size,num_hiddens)\n",
    "        return output, state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss 0.017, 24766.6 tokens/sec on cuda:0\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"262.1875pt\" height=\"183.935303pt\" viewBox=\"0 0 262.1875 183.935303\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2023-05-15T21:06:57.711632</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 183.935303 \n",
       "L 262.1875 183.935303 \n",
       "L 262.1875 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 50.14375 146.379053 \n",
       "L 245.44375 146.379053 \n",
       "L 245.44375 7.779053 \n",
       "L 50.14375 7.779053 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 77.081681 146.379053 \n",
       "L 77.081681 7.779053 \n",
       "\" clip-path=\"url(#p1dafcb9d94)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"m6b48040015\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m6b48040015\" x=\"77.081681\" y=\"146.379053\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 50 -->\n",
       "      <g transform=\"translate(70.719181 160.97749)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 110.754095 146.379053 \n",
       "L 110.754095 7.779053 \n",
       "\" clip-path=\"url(#p1dafcb9d94)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m6b48040015\" x=\"110.754095\" y=\"146.379053\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(101.210345 160.97749)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 144.426509 146.379053 \n",
       "L 144.426509 7.779053 \n",
       "\" clip-path=\"url(#p1dafcb9d94)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m6b48040015\" x=\"144.426509\" y=\"146.379053\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 150 -->\n",
       "      <g transform=\"translate(134.882759 160.97749)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 178.098922 146.379053 \n",
       "L 178.098922 7.779053 \n",
       "\" clip-path=\"url(#p1dafcb9d94)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m6b48040015\" x=\"178.098922\" y=\"146.379053\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 200 -->\n",
       "      <g transform=\"translate(168.555172 160.97749)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 211.771336 146.379053 \n",
       "L 211.771336 7.779053 \n",
       "\" clip-path=\"url(#p1dafcb9d94)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m6b48040015\" x=\"211.771336\" y=\"146.379053\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 250 -->\n",
       "      <g transform=\"translate(202.227586 160.97749)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 245.44375 146.379053 \n",
       "L 245.44375 7.779053 \n",
       "\" clip-path=\"url(#p1dafcb9d94)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m6b48040015\" x=\"245.44375\" y=\"146.379053\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- 300 -->\n",
       "      <g transform=\"translate(235.9 160.97749)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_7\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(132.565625 174.655615)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 50.14375 116.566051 \n",
       "L 245.44375 116.566051 \n",
       "\" clip-path=\"url(#p1dafcb9d94)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <defs>\n",
       "       <path id=\"m24db2fa8ef\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m24db2fa8ef\" x=\"50.14375\" y=\"116.566051\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0.05 -->\n",
       "      <g transform=\"translate(20.878125 120.36527)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n",
       "L 1344 794 \n",
       "L 1344 0 \n",
       "L 684 0 \n",
       "L 684 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"159.033203\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 50.14375 81.377107 \n",
       "L 245.44375 81.377107 \n",
       "\" clip-path=\"url(#p1dafcb9d94)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m24db2fa8ef\" x=\"50.14375\" y=\"81.377107\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 0.10 -->\n",
       "      <g transform=\"translate(20.878125 85.176326)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"95.410156\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"159.033203\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <path d=\"M 50.14375 46.188163 \n",
       "L 245.44375 46.188163 \n",
       "\" clip-path=\"url(#p1dafcb9d94)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m24db2fa8ef\" x=\"50.14375\" y=\"46.188163\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 0.15 -->\n",
       "      <g transform=\"translate(20.878125 49.987382)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"95.410156\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"159.033203\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <path d=\"M 50.14375 10.999219 \n",
       "L 245.44375 10.999219 \n",
       "\" clip-path=\"url(#p1dafcb9d94)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m24db2fa8ef\" x=\"50.14375\" y=\"10.999219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0.20 -->\n",
       "      <g transform=\"translate(20.878125 14.798437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"95.410156\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"159.033203\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_12\">\n",
       "     <!-- loss -->\n",
       "     <g transform=\"translate(14.798437 86.736865)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"27.783203\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"88.964844\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"141.064453\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_21\">\n",
       "    <path d=\"M 50.14375 14.079053 \n",
       "L 56.878233 77.374098 \n",
       "L 63.612716 114.058707 \n",
       "L 70.347198 130.722305 \n",
       "L 77.081681 135.858258 \n",
       "L 83.816164 137.980611 \n",
       "L 90.550647 138.601806 \n",
       "L 97.285129 138.552991 \n",
       "L 104.019612 139.076161 \n",
       "L 110.754095 139.341234 \n",
       "L 117.488578 139.356512 \n",
       "L 124.22306 139.539444 \n",
       "L 130.957543 139.618645 \n",
       "L 137.692026 139.602745 \n",
       "L 144.426509 139.731119 \n",
       "L 151.160991 139.746439 \n",
       "L 157.895474 139.78574 \n",
       "L 164.629957 139.778187 \n",
       "L 171.36444 139.848378 \n",
       "L 178.098922 139.90632 \n",
       "L 184.833405 139.888207 \n",
       "L 191.567888 139.916158 \n",
       "L 198.302371 140.079053 \n",
       "L 205.036853 139.940005 \n",
       "L 211.771336 139.909435 \n",
       "L 218.505819 139.969407 \n",
       "L 225.240302 140.028289 \n",
       "L 231.974784 139.974486 \n",
       "L 238.709267 139.98087 \n",
       "L 245.44375 139.948044 \n",
       "\" clip-path=\"url(#p1dafcb9d94)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 50.14375 146.379053 \n",
       "L 50.14375 7.779053 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 245.44375 146.379053 \n",
       "L 245.44375 7.779053 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 50.14375 146.379053 \n",
       "L 245.44375 146.379053 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 50.14375 7.779053 \n",
       "L 245.44375 7.779053 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p1dafcb9d94\">\n",
       "   <rect x=\"50.14375\" y=\"7.779053\" width=\"195.3\" height=\"138.6\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 350x250 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "embed_size, num_hiddens, num_layers, dropout = 256, 256, 2, 0.1\n",
    "batch_size, num_steps = 256, 10\n",
    "lr, num_epochs, device = 0.001, 300, d2l.try_gpu()\n",
    "\n",
    "train_iter, src_vocab, tgt_vocab = d2l.load_data_nmt(batch_size, num_steps)\n",
    "encoder = d2l.Seq2SeqEncoder(len(src_vocab), embed_size, num_hiddens, num_layers,\n",
    "                        dropout)\n",
    "decoder = Seq2SeqDecoder(len(tgt_vocab), embed_size, num_hiddens, num_layers,\n",
    "                        dropout)\n",
    "net = d2l.EncoderDecoder(encoder, decoder)\n",
    "d2l.train_seq2seq(net, train_iter, lr, num_epochs, tgt_vocab, device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "go . => va !, bleu 1.000\n",
      "i lost . => j'ai perdu ., bleu 1.000\n",
      "he's calm . => il est mouillé mouillé ., bleu 0.548\n",
      "i'm home . => je suis chez <unk> moi qui ai ai <unk> gagné, bleu 0.434\n"
     ]
    }
   ],
   "source": [
    "engs = ['go .', \"i lost .\", 'he\\'s calm .', 'i\\'m home .']\n",
    "fras = ['va !', 'j\\'ai perdu .', 'il est calme .', 'je suis chez moi .']\n",
    "for eng, fra in zip(engs, fras):\n",
    "    translation, attention_weight_seq = d2l.predict_seq2seq(\n",
    "        net, eng, src_vocab, tgt_vocab, num_steps, device)\n",
    "    print(f'{eng} => {translation}, bleu {d2l.bleu(translation, fra, k=2):.3f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.7.2\n",
    "\n",
    "重新运行实验并在计算损失时不使用遮蔽，可以观察到什么结果？为什么会有这个结果？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "class MaskedSoftmaxCELoss(nn.CrossEntropyLoss):\n",
    "    \"\"\"带遮蔽的softmax交叉熵损失函数\"\"\"\n",
    "    # pred的形状：(batch_size,num_steps,vocab_size)\n",
    "    # label的形状：(batch_size,num_steps)\n",
    "    # valid_len的形状：(batch_size,)\n",
    "    def forward(self, pred, label, valid_len):\n",
    "        weights = torch.ones_like(label)\n",
    "        # weights = sequence_mask(weights, valid_len)\n",
    "        self.reduction='none'\n",
    "        unweighted_loss = super(MaskedSoftmaxCELoss, self).forward(\n",
    "            pred.permute(0, 2, 1), label)\n",
    "        weighted_loss = (unweighted_loss * weights).mean(dim=1)\n",
    "        return weighted_loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([2.3026, 2.3026, 2.3026])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss = MaskedSoftmaxCELoss()\n",
    "loss(torch.ones(3, 4, 10), torch.ones((3, 4), dtype=torch.long),\n",
    "     torch.tensor([4, 2, 0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([2.3026, 1.1513, 0.0000])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss_s = d2l.MaskedSoftmaxCELoss()\n",
    "loss_s(torch.ones(3, 4, 10), torch.ones((3, 4), dtype=torch.long),\n",
    "     torch.tensor([4, 2, 0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;观察上面两组值，我们可以发现，如果计算损失时不使用遮蔽，将会输出三个相同的序列损失值，而非出现第一个序列的损失为第二个序列的两倍，第三个序列的损失为零。  \n",
    "&emsp;&emsp;出现这种情况的原因在于，注释掉遮蔽函数 **sequence_mask()** 后，无法再根据有效长度 **valid_len** 对权重 **weights** 进行遮蔽操作，使得所有位置的权重都将参与计算，而非仅限于有效长度内的位置。这可能会导致在计算损失时，对序列中无效的部分产生误差，影响模型的训练效果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.7.3\n",
    "\n",
    "如果编码器和解码器的层数或者隐藏单元数不同，那么如何初始化解码器的隐状态？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;如果编码器和解码器的层数或隐藏单元数不同，可以采取以下方法来初始化解码器的隐状态：\n",
    "- 使用编码器隐藏状态的线性变换：如果编码器和解码器的隐藏单元数不同，可以通过线性变换将编码器隐藏状态的维度调整为解码器隐藏状态所需的维度。可以使用全连接层或线性映射来实现这个变换。\n",
    "- 使用零向量进行初始化：如果编码器和解码器的隐藏单元数差异较大，或者无法进行有效的维度变换，可以使用零向量来初始化解码器的隐状态。这种方法可能会导致一开始解码器的性能较差，但随着训练的进行，解码器可以逐渐学习到适应目标任务的隐状态表示。\n",
    "\n",
    "&emsp;&emsp;下面是引入全连接层进行修改的示例："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "class Seq2SeqDecoder(d2l.Decoder):\n",
    "    \"\"\"用于序列到序列学习的循环神经网络解码器\"\"\"\n",
    "    def __init__(self, vocab_size, embed_size, num_hiddens_enc, num_hiddens_dec, num_layers,\n",
    "                 dropout=0, **kwargs):\n",
    "        super(Seq2SeqDecoder, self).__init__(**kwargs)\n",
    "        self.embedding = nn.Embedding(vocab_size, embed_size)\n",
    "        self.rnn = nn.GRU(embed_size + num_hiddens_dec, num_hiddens_dec, num_layers,\n",
    "                          dropout=dropout)\n",
    "        self.fc = nn.Linear(num_hiddens_enc, num_hiddens_dec)  # 添加全连接层\n",
    "        self.dense = nn.Linear(num_hiddens_dec, vocab_size)\n",
    "\n",
    "    def init_state(self, enc_outputs, *args):\n",
    "        return self.fc(enc_outputs[1])  # 使用全连接层调整隐藏状态的维度\n",
    "\n",
    "    def forward(self, X, state):\n",
    "        # 输出'X'的形状：(batch_size, num_steps, embed_size)\n",
    "        X = self.embedding(X).permute(1, 0, 2)\n",
    "        # 广播context，使其具有与X相同的num_steps\n",
    "        context = state[-1].repeat(X.shape[0], 1, 1)\n",
    "        X_and_context = torch.cat((X, context), 2)\n",
    "\n",
    "        output, state = self.rnn(X_and_context, state)\n",
    "        output = self.dense(output).permute(1, 0, 2)\n",
    "        # output的形状: (batch_size, num_steps, vocab_size)\n",
    "        # state的形状: (num_layers, batch_size, num_hiddens_dec)\n",
    "        return output, state\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([7, 4, 16])"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoder = d2l.Seq2SeqEncoder(vocab_size=10, embed_size=8, num_hiddens=16,\n",
    "                         num_layers=2)\n",
    "encoder.eval()\n",
    "X = torch.zeros((4, 7), dtype=torch.long)\n",
    "output, state = encoder(X)\n",
    "output.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([4, 7, 10]), torch.Size([2, 4, 10]))"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decoder = Seq2SeqDecoder(vocab_size=10, embed_size=8, num_hiddens_enc=16, num_hiddens_dec=10,\n",
    "                         num_layers=2)\n",
    "decoder.eval()\n",
    "state = decoder.init_state(encoder(X))\n",
    "output, state = decoder(X, state)\n",
    "output.shape, state.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "###  练习 9.7.4\n",
    "\n",
    "在训练中，如果用前一时间步的预测输入到解码器来代替强制教学，对性能有何影响？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;在训练中，这种采用前一时间步的预测作为解码器输入的方法属于 **自回归** 方法，这种自回归的方法，对于性能会产生以下影响：\n",
    "- 训练速度变慢：使用自回归训练方式会导致解码器逐步生成输出序列，而不是一次性使用整个目标序列。这会增加训练的时间，因为需要进行逐步的预测和反向传播。\n",
    "- 累积误差：由于解码器的输入是前一时间步的预测结果，而不是真实目标序列，错误可能会逐步累积并扩大。这可能导致生成的序列与真实目标序列之间存在较大的差异。\n",
    "- 降低训练稳定性：自回归训练方式可能会导致训练过程更不稳定，特别是在序列较长、任务复杂的情况下。错误的累积和误差传播可能会导致训练困难，并且模型可能会在某些时间步骤上产生不合理的输出。\n",
    "- 降低性能：自回归训练方式可能会降低模型在测试集或实际应用中的性能。由于解码器在训练期间没有接触到真实的目标序列，它可能无法很好地适应真实数据的分布和模式。\n",
    "\n",
    "&emsp;&emsp;即便如此，目前自回归方法仍被广泛应用于序列生成任务中，通过引入如注意力机制等改进方法后，模型可以有效提升性能，生成拟合度更好的序列。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "#@save\n",
    "def train_seq2seq(net, data_iter, lr, num_epochs, tgt_vocab, device):\n",
    "    \"\"\"训练序列到序列模型\"\"\"\n",
    "    def xavier_init_weights(m):\n",
    "        if type(m) == nn.Linear:\n",
    "            nn.init.xavier_uniform_(m.weight)\n",
    "        if type(m) == nn.GRU:\n",
    "            for param in m._flat_weights_names:\n",
    "                if \"weight\" in param:\n",
    "                    nn.init.xavier_uniform_(m._parameters[param])\n",
    "\n",
    "    net.apply(xavier_init_weights)\n",
    "    net.to(device)\n",
    "    optimizer = torch.optim.Adam(net.parameters(), lr=lr)\n",
    "    loss = MaskedSoftmaxCELoss()\n",
    "    net.train()\n",
    "    animator = d2l.Animator(xlabel='epoch', ylabel='loss',\n",
    "                     xlim=[10, num_epochs])\n",
    "    for epoch in range(num_epochs):\n",
    "        timer = d2l.Timer()\n",
    "        metric = d2l.Accumulator(2)  # 训练损失总和，词元数量\n",
    "        for batch in data_iter:\n",
    "            optimizer.zero_grad()\n",
    "            X, X_valid_len, Y, Y_valid_len = [x.to(device) for x in batch]\n",
    "            bos = torch.tensor([tgt_vocab['<bos>']] * Y.shape[0],\n",
    "                          device=device).reshape(-1, 1)\n",
    "            # dec_input = torch.cat([bos, Y[:, :-1]], 1)  # 强制教学\n",
    "\n",
    "            dec_input = bos\n",
    "            state = None\n",
    "            output_seq = []\n",
    "            for i in range(Y.shape[1]):\n",
    "                Y_hat, state = net(X, dec_input, X_valid_len, state)    # 将<bos>输入，获得预测结果\n",
    "                dec_input = Y_hat.argmax(dim=-1)  # 使用模型预测的结果作为下一个时间步的输入\n",
    "                output_seq.append(Y_hat)\n",
    "            output_seq = torch.cat(output_seq, dim=1)       # 拼接所有预测结果，获得完整序列\n",
    "            Y_hat = output_seq\n",
    "\n",
    "            # Y_hat, _ = net(X, dec_input, X_valid_len)\n",
    "            l = loss(Y_hat, Y, Y_valid_len)\n",
    "            l.sum().backward()      # 损失函数的标量进行“反向传播”\n",
    "            d2l.grad_clipping(net, 1)\n",
    "            num_tokens = Y_valid_len.sum()\n",
    "            optimizer.step()\n",
    "            with torch.no_grad():\n",
    "                metric.add(l.sum(), num_tokens)\n",
    "        if (epoch + 1) % 10 == 0:\n",
    "            animator.add(epoch + 1, (metric[0] / metric[1],))\n",
    "    print(f'loss {metric[0] / metric[1]:.3f}, {metric[1] / timer.stop():.1f} '\n",
    "        f'tokens/sec on {str(device)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "class Seq2SeqDecoder(d2l.Decoder):\n",
    "    \"\"\"用于序列到序列学习的循环神经网络解码器\"\"\"\n",
    "    def __init__(self, vocab_size, embed_size, num_hiddens, num_layers,\n",
    "                 dropout=0, **kwargs):\n",
    "        super(Seq2SeqDecoder, self).__init__(**kwargs)\n",
    "        self.embedding = nn.Embedding(vocab_size, embed_size)\n",
    "        self.rnn = nn.GRU(embed_size + num_hiddens, num_hiddens, num_layers,\n",
    "                          dropout=dropout)\n",
    "        self.dense = nn.Linear(num_hiddens, vocab_size)\n",
    "\n",
    "    def init_state(self, enc_outputs, *args):\n",
    "        return enc_outputs[1]\n",
    "\n",
    "    def forward(self, X, state):\n",
    "        # 输出'X'的形状：(batch_size,num_steps,embed_size)\n",
    "        X = self.embedding(X).permute(1, 0, 2)\n",
    "        # 广播context，使其具有与X相同的num_steps\n",
    "        context = state[-1].repeat(X.shape[0], 1, 1)\n",
    "        X_and_context = torch.cat((X, context), 2)\n",
    "        output, state = self.rnn(X_and_context, state)\n",
    "        output = self.dense(output).permute(1, 0, 2)\n",
    "        # output的形状:(batch_size,num_steps,vocab_size)\n",
    "        # state的形状:(num_layers,batch_size,num_hiddens)\n",
    "        return output, state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss 0.064, 9132.0 tokens/sec on cuda:0\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"255.825pt\" height=\"183.35625pt\" viewBox=\"0 0 255.825 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2023-05-17T16:06:03.858254</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 183.35625 \n",
       "L 255.825 183.35625 \n",
       "L 255.825 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 43.78125 145.8 \n",
       "L 239.08125 145.8 \n",
       "L 239.08125 7.2 \n",
       "L 43.78125 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 70.719181 145.8 \n",
       "L 70.719181 7.2 \n",
       "\" clip-path=\"url(#p53dbb3af75)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"m03d3a70096\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m03d3a70096\" x=\"70.719181\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 50 -->\n",
       "      <g transform=\"translate(64.356681 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 104.391595 145.8 \n",
       "L 104.391595 7.2 \n",
       "\" clip-path=\"url(#p53dbb3af75)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m03d3a70096\" x=\"104.391595\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(94.847845 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 138.064009 145.8 \n",
       "L 138.064009 7.2 \n",
       "\" clip-path=\"url(#p53dbb3af75)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m03d3a70096\" x=\"138.064009\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 150 -->\n",
       "      <g transform=\"translate(128.520259 160.398438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 171.736422 145.8 \n",
       "L 171.736422 7.2 \n",
       "\" clip-path=\"url(#p53dbb3af75)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m03d3a70096\" x=\"171.736422\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 200 -->\n",
       "      <g transform=\"translate(162.192672 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 205.408836 145.8 \n",
       "L 205.408836 7.2 \n",
       "\" clip-path=\"url(#p53dbb3af75)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m03d3a70096\" x=\"205.408836\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 250 -->\n",
       "      <g transform=\"translate(195.865086 160.398438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 239.08125 145.8 \n",
       "L 239.08125 7.2 \n",
       "\" clip-path=\"url(#p53dbb3af75)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m03d3a70096\" x=\"239.08125\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- 300 -->\n",
       "      <g transform=\"translate(229.5375 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_7\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(126.203125 174.076563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 43.78125 125.073992 \n",
       "L 239.08125 125.073992 \n",
       "\" clip-path=\"url(#p53dbb3af75)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <defs>\n",
       "       <path id=\"m9c5becee9b\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m9c5becee9b\" x=\"43.78125\" y=\"125.073992\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0.1 -->\n",
       "      <g transform=\"translate(20.878125 128.87321)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n",
       "L 1344 794 \n",
       "L 1344 0 \n",
       "L 684 0 \n",
       "L 684 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 43.78125 89.08548 \n",
       "L 239.08125 89.08548 \n",
       "\" clip-path=\"url(#p53dbb3af75)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m9c5becee9b\" x=\"43.78125\" y=\"89.08548\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 0.2 -->\n",
       "      <g transform=\"translate(20.878125 92.884699)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <path d=\"M 43.78125 53.096969 \n",
       "L 239.08125 53.096969 \n",
       "\" clip-path=\"url(#p53dbb3af75)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m9c5becee9b\" x=\"43.78125\" y=\"53.096969\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 0.3 -->\n",
       "      <g transform=\"translate(20.878125 56.896187)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-33\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <path d=\"M 43.78125 17.108457 \n",
       "L 239.08125 17.108457 \n",
       "\" clip-path=\"url(#p53dbb3af75)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m9c5becee9b\" x=\"43.78125\" y=\"17.108457\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0.4 -->\n",
       "      <g transform=\"translate(20.878125 20.907676)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-34\" x=\"95.410156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_12\">\n",
       "     <!-- loss -->\n",
       "     <g transform=\"translate(14.798437 86.157813)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"27.783203\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"88.964844\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"141.064453\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_21\">\n",
       "    <path d=\"M 43.78125 13.5 \n",
       "L 50.515733 56.970019 \n",
       "L 57.250216 73.607308 \n",
       "L 63.984698 78.56549 \n",
       "L 70.719181 90.915689 \n",
       "L 77.453664 98.22586 \n",
       "L 84.188147 108.723806 \n",
       "L 90.922629 113.728952 \n",
       "L 97.657112 117.256924 \n",
       "L 104.391595 122.333696 \n",
       "L 111.126078 125.338387 \n",
       "L 117.86056 127.211944 \n",
       "L 124.595043 128.362723 \n",
       "L 131.329526 131.379269 \n",
       "L 138.064009 130.750375 \n",
       "L 144.798491 133.619665 \n",
       "L 151.532974 133.607685 \n",
       "L 158.267457 133.564962 \n",
       "L 165.00194 135.50513 \n",
       "L 171.736422 137.071042 \n",
       "L 178.470905 136.116452 \n",
       "L 185.205388 137.023343 \n",
       "L 191.939871 137.420267 \n",
       "L 198.674353 136.695541 \n",
       "L 205.408836 138.625249 \n",
       "L 212.143319 137.837018 \n",
       "L 218.877802 137.496121 \n",
       "L 225.612284 139.5 \n",
       "L 232.346767 136.546138 \n",
       "L 239.08125 138.178404 \n",
       "\" clip-path=\"url(#p53dbb3af75)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 43.78125 145.8 \n",
       "L 43.78125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 239.08125 145.8 \n",
       "L 239.08125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 43.78125 145.8 \n",
       "L 239.08125 145.8 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 43.78125 7.2 \n",
       "L 239.08125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p53dbb3af75\">\n",
       "   <rect x=\"43.78125\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 350x250 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "embed_size, num_hiddens, num_layers, dropout = 32, 32, 2, 0.1\n",
    "batch_size, num_steps = 64, 10\n",
    "lr, num_epochs, device = 0.005, 300, d2l.try_gpu()\n",
    "\n",
    "train_iter, src_vocab, tgt_vocab = d2l.load_data_nmt(batch_size, num_steps)\n",
    "encoder = d2l.Seq2SeqEncoder(len(src_vocab), embed_size, num_hiddens, num_layers,\n",
    "                        dropout)\n",
    "decoder = Seq2SeqDecoder(len(tgt_vocab), embed_size, num_hiddens, num_layers,\n",
    "                        dropout)\n",
    "net = d2l.EncoderDecoder(encoder, decoder)\n",
    "train_seq2seq(net, train_iter, lr, num_epochs, tgt_vocab, device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "go . => va va ne va ne trouve trouve trouve trouve trouve, bleu 0.000\n",
      "i lost . => j'ai je je tu tu tu tu tu tu tu, bleu 0.000\n",
      "he's calm . => c'est comme comme comme comme comme comme comme comme comme, bleu 0.000\n",
      "i'm home . => je je je je je je je je je je, bleu 0.000\n"
     ]
    }
   ],
   "source": [
    "engs = ['go .', \"i lost .\", 'he\\'s calm .', 'i\\'m home .']\n",
    "fras = ['va !', 'j\\'ai perdu .', 'il est calme .', 'je suis chez moi .']\n",
    "for eng, fra in zip(engs, fras):\n",
    "    translation, attention_weight_seq = d2l.predict_seq2seq(\n",
    "        net, eng, src_vocab, tgt_vocab, num_steps, device)\n",
    "    print(f'{eng} => {translation}, bleu {d2l.bleu(translation, fra, k=2):.3f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.7.5\n",
    "\n",
    "用长短期记忆网络替换门控循环单元重新运行实验。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "#@save\n",
    "class Seq2SeqEncoder(d2l.Encoder):\n",
    "    \"\"\"用于序列到序列学习的循环神经网络编码器\"\"\"\n",
    "    def __init__(self, vocab_size, embed_size, num_hiddens, num_layers,\n",
    "                 dropout=0, **kwargs):\n",
    "        super(Seq2SeqEncoder, self).__init__(**kwargs)\n",
    "        # 嵌入层\n",
    "        self.embedding = nn.Embedding(vocab_size, embed_size)\n",
    "        self.rnn = nn.LSTM(embed_size, num_hiddens, num_layers,\n",
    "                          dropout=dropout)\n",
    "\n",
    "    def forward(self, X, *args):\n",
    "        # 输出'X'的形状：(batch_size,num_steps,embed_size)\n",
    "        X = self.embedding(X)\n",
    "        # 在循环神经网络模型中，第一个轴对应于时间步\n",
    "        X = X.permute(1, 0, 2)\n",
    "        # 如果未提及状态，则默认为0\n",
    "        output, state = self.rnn(X)\n",
    "        # output的形状:(num_steps,batch_size,num_hiddens)\n",
    "        # state的形状:(num_layers,batch_size,num_hiddens)\n",
    "        return output, state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([7, 4, 16])"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoder = Seq2SeqEncoder(vocab_size=10, embed_size=8, num_hiddens=16,\n",
    "                         num_layers=2)\n",
    "encoder.eval()\n",
    "X = torch.zeros((4, 7), dtype=torch.long)\n",
    "output, state = encoder(X)\n",
    "output.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 4, 16])"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "state[0].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "class Seq2SeqDecoder(d2l.Decoder):\n",
    "    \"\"\"用于序列到序列学习的循环神经网络解码器\"\"\"\n",
    "    def __init__(self, vocab_size, embed_size, num_hiddens, num_layers,\n",
    "                 dropout=0, **kwargs):\n",
    "        super(Seq2SeqDecoder, self).__init__(**kwargs)\n",
    "        self.embedding = nn.Embedding(vocab_size, embed_size)\n",
    "        self.rnn = nn.LSTM(embed_size + num_hiddens, num_hiddens, num_layers,\n",
    "                          dropout=dropout)\n",
    "        self.dense = nn.Linear(num_hiddens, vocab_size)\n",
    "\n",
    "    def init_state(self, enc_outputs, *args):\n",
    "        state = enc_outputs[1]\n",
    "        return state\n",
    "\n",
    "    def forward(self, X, state):\n",
    "        # 输出'X'的形状：(batch_size,num_steps,embed_size)\n",
    "        X = self.embedding(X).permute(1, 0, 2)\n",
    "        # 广播context，使其具有与X相同的num_steps\n",
    "        context = state[-1][0].unsqueeze(0).repeat(X.shape[0], 1, 1)    # 在LSTM中，相较于隐藏状态，记忆元中存储了更多信息，有利于捕捉长期依赖关系，因此这里提取记忆元而非隐藏状态\n",
    "        X_and_context = torch.cat((X, context), 2)\n",
    "        output, state = self.rnn(X_and_context, state)\n",
    "        output = self.dense(output).permute(1, 0, 2)\n",
    "        # output的形状:(batch_size,num_steps,vocab_size)\n",
    "        # state的形状:(num_layers,batch_size,num_hiddens)\n",
    "        return output, state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([4, 7, 10]), torch.Size([2, 4, 16]))"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decoder = Seq2SeqDecoder(vocab_size=10, embed_size=8, num_hiddens=16,\n",
    "                         num_layers=2)\n",
    "decoder.eval()\n",
    "state = decoder.init_state(encoder(X))\n",
    "\n",
    "output, state = decoder(X, state)\n",
    "output.shape, state[0].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss 0.018, 19088.3 tokens/sec on cuda:0\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"262.1875pt\" height=\"183.35625pt\" viewBox=\"0 0 262.1875 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2023-05-17T11:34:28.571026</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 183.35625 \n",
       "L 262.1875 183.35625 \n",
       "L 262.1875 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 50.14375 145.8 \n",
       "L 245.44375 145.8 \n",
       "L 245.44375 7.2 \n",
       "L 50.14375 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path d=\"M 86.015179 145.8 \n",
       "L 86.015179 7.2 \n",
       "\" clip-path=\"url(#pa9b21d4a16)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path id=\"m59c3ac1b38\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m59c3ac1b38\" x=\"86.015179\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(76.471429 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path d=\"M 125.872321 145.8 \n",
       "L 125.872321 7.2 \n",
       "\" clip-path=\"url(#pa9b21d4a16)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m59c3ac1b38\" x=\"125.872321\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 200 -->\n",
       "      <g transform=\"translate(116.328571 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path d=\"M 165.729464 145.8 \n",
       "L 165.729464 7.2 \n",
       "\" clip-path=\"url(#pa9b21d4a16)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m59c3ac1b38\" x=\"165.729464\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 300 -->\n",
       "      <g transform=\"translate(156.185714 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path d=\"M 205.586607 145.8 \n",
       "L 205.586607 7.2 \n",
       "\" clip-path=\"url(#pa9b21d4a16)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m59c3ac1b38\" x=\"205.586607\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 400 -->\n",
       "      <g transform=\"translate(196.042857 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path d=\"M 245.44375 145.8 \n",
       "L 245.44375 7.2 \n",
       "\" clip-path=\"url(#pa9b21d4a16)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m59c3ac1b38\" x=\"245.44375\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 500 -->\n",
       "      <g transform=\"translate(235.9 160.398438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(132.565625 174.076563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path d=\"M 50.14375 121.691187 \n",
       "L 245.44375 121.691187 \n",
       "\" clip-path=\"url(#pa9b21d4a16)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path id=\"md9c062b93b\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#md9c062b93b\" x=\"50.14375\" y=\"121.691187\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0.05 -->\n",
       "      <g transform=\"translate(20.878125 125.490406)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n",
       "L 1344 794 \n",
       "L 1344 0 \n",
       "L 684 0 \n",
       "L 684 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"159.033203\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path d=\"M 50.14375 94.023799 \n",
       "L 245.44375 94.023799 \n",
       "\" clip-path=\"url(#pa9b21d4a16)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#md9c062b93b\" x=\"50.14375\" y=\"94.023799\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0.10 -->\n",
       "      <g transform=\"translate(20.878125 97.823018)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"95.410156\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"159.033203\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path d=\"M 50.14375 66.356412 \n",
       "L 245.44375 66.356412 \n",
       "\" clip-path=\"url(#pa9b21d4a16)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#md9c062b93b\" x=\"50.14375\" y=\"66.356412\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 0.15 -->\n",
       "      <g transform=\"translate(20.878125 70.15563)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"95.410156\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"159.033203\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <path d=\"M 50.14375 38.689024 \n",
       "L 245.44375 38.689024 \n",
       "\" clip-path=\"url(#pa9b21d4a16)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#md9c062b93b\" x=\"50.14375\" y=\"38.689024\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 0.20 -->\n",
       "      <g transform=\"translate(20.878125 42.488243)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"95.410156\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"159.033203\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <path d=\"M 50.14375 11.021636 \n",
       "L 245.44375 11.021636 \n",
       "\" clip-path=\"url(#pa9b21d4a16)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#md9c062b93b\" x=\"50.14375\" y=\"11.021636\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0.25 -->\n",
       "      <g transform=\"translate(20.878125 14.820855)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"95.410156\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-35\" x=\"159.033203\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_12\">\n",
       "     <!-- loss -->\n",
       "     <g transform=\"translate(14.798437 86.157813)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"27.783203\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"88.964844\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"141.064453\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_21\">\n",
       "    <path d=\"M 50.14375 13.5 \n",
       "L 54.129464 37.745851 \n",
       "L 58.115179 57.435968 \n",
       "L 62.100893 69.522812 \n",
       "L 66.086607 79.577803 \n",
       "L 70.072321 88.318655 \n",
       "L 74.058036 95.695252 \n",
       "L 78.04375 102.061551 \n",
       "L 82.029464 107.84766 \n",
       "L 86.015179 112.616651 \n",
       "L 90.000893 117.355982 \n",
       "L 93.986607 120.585343 \n",
       "L 97.972321 123.691381 \n",
       "L 101.958036 126.614055 \n",
       "L 105.94375 128.613806 \n",
       "L 109.929464 130.338616 \n",
       "L 113.915179 132.084358 \n",
       "L 117.900893 133.316953 \n",
       "L 121.886607 133.920947 \n",
       "L 125.872321 134.956866 \n",
       "L 129.858036 135.668811 \n",
       "L 133.84375 136.041295 \n",
       "L 137.829464 136.687854 \n",
       "L 141.815179 137.107003 \n",
       "L 145.800893 137.273388 \n",
       "L 149.786607 137.546312 \n",
       "L 153.772321 137.937475 \n",
       "L 157.758036 137.955782 \n",
       "L 161.74375 138.116371 \n",
       "L 165.729464 138.325434 \n",
       "L 169.715179 138.594169 \n",
       "L 173.700893 138.461603 \n",
       "L 177.686607 138.69953 \n",
       "L 181.672321 138.977745 \n",
       "L 185.658036 138.735408 \n",
       "L 189.64375 138.704965 \n",
       "L 193.629464 138.843663 \n",
       "L 197.615179 138.879121 \n",
       "L 201.600893 139.017696 \n",
       "L 205.586607 139.283059 \n",
       "L 209.572321 139.046512 \n",
       "L 213.558036 139.061358 \n",
       "L 217.54375 139.266415 \n",
       "L 221.529464 139.174269 \n",
       "L 225.515179 139.28117 \n",
       "L 229.500893 139.082741 \n",
       "L 233.486607 139.234137 \n",
       "L 237.472321 139.432179 \n",
       "L 241.458036 139.5 \n",
       "L 245.44375 139.294694 \n",
       "\" clip-path=\"url(#pa9b21d4a16)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 50.14375 145.8 \n",
       "L 50.14375 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 245.44375 145.8 \n",
       "L 245.44375 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 50.14375 145.8 \n",
       "L 245.44375 145.8 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 50.14375 7.2 \n",
       "L 245.44375 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pa9b21d4a16\">\n",
       "   <rect x=\"50.14375\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 350x250 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "embed_size, num_hiddens, num_layers, dropout = 32, 32, 2, 0.1\n",
    "batch_size, num_steps = 64, 10\n",
    "lr, num_epochs, device = 0.005, 500, d2l.try_gpu()\n",
    "\n",
    "train_iter, src_vocab, tgt_vocab = d2l.load_data_nmt(batch_size, num_steps)\n",
    "encoder = Seq2SeqEncoder(len(src_vocab), embed_size, num_hiddens, num_layers,\n",
    "                        dropout)\n",
    "decoder = Seq2SeqDecoder(len(tgt_vocab), embed_size, num_hiddens, num_layers,\n",
    "                        dropout)\n",
    "net = d2l.EncoderDecoder(encoder, decoder)\n",
    "d2l.train_seq2seq(net, train_iter, lr, num_epochs, tgt_vocab, device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "go . => va !, bleu 1.000\n",
      "i lost . => j'ai perdu ., bleu 1.000\n",
      "he's calm . => il est calme ., bleu 1.000\n",
      "i'm home . => je suis chez moi ., bleu 1.000\n"
     ]
    }
   ],
   "source": [
    "engs = ['go .', \"i lost .\", 'he\\'s calm .', 'i\\'m home .']\n",
    "fras = ['va !', 'j\\'ai perdu .', 'il est calme .', 'je suis chez moi .']\n",
    "for eng, fra in zip(engs, fras):\n",
    "    translation, attention_weight_seq = d2l.predict_seq2seq(\n",
    "        net, eng, src_vocab, tgt_vocab, num_steps, device)\n",
    "    print(f'{eng} => {translation}, bleu {d2l.bleu(translation, fra, k=2):.3f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.7.6\n",
    "\n",
    "有没有其他方法来设计解码器的输出层？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;可以在解码器的输出层应用注意力机制，通过计算注意力权重来对编码器的隐藏状态进行加权求和，从而生成输出。在类似机器翻译任务中，这种方法可以将解码器的注意力集中在输入序列的相关部分，有效捕捉重要信息。提高生成的准确性。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## 9.8 束搜索 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.8.1\n",
    "\n",
    "我们可以把穷举搜索看作一种特殊的束搜索吗？为什么？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;穷举与束搜索采用了不同的思路，并不能将其看作是特殊的束搜索。  \n",
    "&emsp;&emsp;在进行穷举时，算法会对解空间中的所有解进行遍历搜索，然后计算出最优结果。因此导致计算量巨大。但是束搜索则是根据设置好的束宽$K$来进行搜索，在每一步选择$K$个最优结果，在保证找到最优解的前提下，尽量减少空间与时间复杂度。实际上，贪心和束搜索都属于启发式算法的一种，而穷举搜索并不包含任何启发式信息，因此不能看作是特殊的束搜搜。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.8.2\n",
    "\n",
    "在9.7节中的机器翻译问题中应用束搜索。束宽是如何影响预测的速度和结果的？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;在机器翻译任务中，虽然应用束搜索可以有效提高预测的准确性，生成更加合理的翻译结果。但是束搜索也会通过束宽来影响模型的预测速度和结果。  \n",
    "&emsp;&emsp;作为控制候选项数量的参数，束宽较小时会减少搜索空间，带来更快的预测速度。而在较大时会保留更多的候选项，虽然可以带来更加准确的结果，但是也会增加计算的复杂性和预测时间。因此在设置束宽时需要权衡预测速度和结果精度，根据实际需求和性能进行调整，以取得满意的翻译结果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### 练习 9.8.3\n",
    "\n",
    "在8.5节中，我们基于用户提供的前缀，通过使用语言模型来生成文本。这个例子中使用了哪种搜索策略？可以改进吗？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "&emsp;&emsp;在8.5节中，最后在生成文本时，在每一步预测中，使用 **y.argmax(dim=1)** 选择具有最高概率的字符作为下一个预测字符。这里的选择是基于当前时刻模型的输出概率最大化的策略，而没有考虑其他备选字符,这种搜索算法实质上属于贪心算法。  \n",
    "&emsp;&emsp;可以通过修改策略来生成更具多样性的文本，例如通过束搜索的方式，在每一步选择多个候选字符，并根据概率得分进行排序。最终只保留得分最高的一部分候选字符，然后基于这些候选字符继续生成下一个字符。这种束搜索算法可以提供更好的生成文本质量和连贯性，但会增加计算复杂度。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
